Method, device and storage medium for determining lyrics time

Abstract

The present application discloses a method, device and storage medium for determining lyrics timestamp, and belongs to the field of speech recognition technology. The method comprises: for each of the pre-trained multiple speech recognition models, inputting the audio features of the target song into the speech recognition model, and obtaining the confidence of the reference phoneme state corresponding to each frame of the audio feature in the language to which the speech recognition model belongs; for each frame of audio features, taking the reference phoneme state with the maximum confidence corresponding to the audio feature as the aligned phoneme state corresponding to the audio feature; for each frame of audio features, determining the phoneme corresponding to the audio feature according to the aligned phoneme state corresponding to the audio feature; determining the word corresponding to each frame of audio features in the lyrics text according to the phoneme corresponding to each frame of audio features, and determining the lyrics timestamp of the lyrics text according to the singing time corresponding to each frame of audio features. The present application can improve the efficiency of determining lyrics timestamps.

Description

Technical Field

[0001] The present application relates to the field of speech recognition technology, and in particular to a method, device and storage medium for determining lyrics time. Background Art

[0002] When a music player application plays a song, the song playback interface can usually display the lyrics corresponding to the current song playback progress. To achieve this effect, lyrics text with lyrics timestamps is required. The lyrics timestamp includes the start time point and the end time point of each word in the lyrics text.

[0003] Currently, the timestamps of lyrics in lyrics files are usually obtained through manual annotation. However, a large number of songs are added to the music library corresponding to the music playing application every day. If the timestamps of lyrics of each song are obtained only through manual annotation, the efficiency is low. Summary of the invention

[0004] The embodiments of the present application provide a method, device and storage medium for determining lyrics time, which can solve the problem of low efficiency in determining lyrics timestamp in related technologies. The technical solution is as follows:

[0005] In a first aspect, a method for determining a lyrics timestamp is provided, the method comprising:

[0006] Obtaining lyrics text of a target song, and extracting audio features of each audio frame in the target song;

[0007] Acquire a plurality of pre-trained speech recognition models, wherein the plurality of speech recognition models are speech recognition models trained with song data samples in different languages;

[0008] For each of the speech recognition models, the audio features of each audio frame in the target song are input into the speech recognition model to obtain the reference phoneme state corresponding to each frame of the audio features in the language of the speech recognition model and the confidence of the reference phoneme state;

[0009] For each frame of the audio feature, taking a reference phoneme state corresponding to a maximum confidence level among multiple confidence levels of the audio feature as an aligned phoneme state of the audio feature, and determining a phoneme corresponding to the audio feature according to the aligned phoneme state of the audio feature;

[0010] The lyrics timestamp of the lyrics text is determined according to the phonemes and singing time corresponding to the audio features of each frame.

[0011] In a possible implementation, the step of inputting the audio features of each audio frame in the target song into the speech recognition model to obtain the reference phoneme state corresponding to each frame of the audio features in the language of the speech recognition model and the confidence of the reference phoneme state includes:

[0012] Input the audio features of each audio frame in the target song and the phoneme sequence of the lyrics text into the speech recognition model, and obtain the reference phoneme state corresponding to the audio features of each frame output by the speech recognition model in the phoneme sequence and the first probability corresponding to the reference phoneme state;

[0013] Inputting the audio features of each audio frame in the target song and the phoneme set of the language to which the speech recognition model belongs into the speech recognition model, and obtaining a second probability that the audio features of each frame output by the speech recognition model correspond to the reference phoneme state in the phoneme set;

[0014] For each frame of audio features, the confidence that the audio features correspond to the reference phoneme states in the language of the speech recognition model is determined based on a first probability and a second probability that the audio features correspond to the reference phoneme states.

[0015] In a possible implementation, the step of inputting the audio features of each audio frame in the target song and the phoneme sequence of the lyrics text into the speech recognition model, and obtaining the reference phoneme state corresponding to the audio features of each frame output by the speech recognition model in the phoneme sequence and the first probability corresponding to the reference phoneme state includes:

[0016] Input the audio features of each audio frame in the target song and the phoneme sequence of the lyrics text into the speech recognition model, and obtain the probability of the audio features of each frame output by the speech recognition model corresponding to each phoneme state in the phoneme sequence;

[0017] For each of the audio features, the phoneme state corresponding to the maximum probability of the audio feature is used as the reference phoneme state corresponding to the audio feature, and the maximum probability is determined as the first probability that the audio feature corresponds to the reference phoneme state.

[0018] In a possible implementation, the step of inputting the audio features of each audio frame in the target song and the phoneme set of the language to which the speech recognition model belongs into the speech recognition model, and obtaining a second probability that the audio features of each frame output by the speech recognition model correspond to the reference phoneme state in the phoneme set includes:

[0019] Inputting the audio features of each audio frame in the target song and the phoneme set of the language to which the speech recognition model belongs into the speech recognition model, and obtaining the probability of each phoneme state in the phoneme set corresponding to the audio features of each frame output by the speech recognition model;

[0020] Among the probabilities of the audio feature corresponding to each phoneme state, the probability of the audio feature corresponding to the reference phoneme state is obtained as the second probability.

[0021] In a possible implementation, determining, based on the first probability and the second probability that the audio feature corresponds to the reference phoneme state, the confidence that the audio feature corresponds to the reference phoneme state in the language of the speech recognition model includes:

[0022] The first probability that the audio feature corresponds to the reference phoneme state is divided by the second probability that the audio feature corresponds to the reference phoneme state to obtain the confidence that the audio feature corresponds to the reference phoneme state in the language of the speech recognition model.

[0023] In a possible implementation, the method further includes:

[0024] The phoneme sequence of the lyrics text is determined according to a mixed pronunciation dictionary; wherein the mixed pronunciation dictionary is composed of pronunciation dictionaries of the languages ​​to which the multiple speech recognition models belong.

[0025] In a possible implementation, determining the phoneme corresponding to the audio feature according to the aligned phoneme state of the audio feature includes:

[0026] According to the correspondence between the phoneme state and the phoneme, the phoneme corresponding to the aligned phoneme state of the audio feature is determined as the phoneme corresponding to the audio feature.

[0027] In a possible implementation, obtaining the lyrics text of the target song includes:

[0028] Obtaining the initial lyrics text of the target song;

[0029] The initial lyrics text is processed by filtering out non-lyrics information to obtain the lyrics text of the target song.

[0030] In a second aspect, a device for determining a lyrics timestamp is provided, the device comprising:

[0031] An acquisition module, used to acquire the lyrics text of a target song and extract audio features of each audio frame in the target song;

[0032] A recognition module is used to obtain a plurality of pre-trained speech recognition models, wherein the plurality of speech recognition models are speech recognition models trained with song data samples in different languages ​​respectively; for each of the speech recognition models, the audio features of each audio frame in the target song are input into the speech recognition model to obtain a reference phoneme state corresponding to each frame of the audio features in the language of the speech recognition model and a confidence level of the reference phoneme state;

[0033] an alignment module, configured to, for each frame of the audio feature, use a reference phoneme state corresponding to a maximum confidence level among a plurality of confidence levels of the audio feature as an aligned phoneme state of the audio feature, and determine a phoneme corresponding to the audio feature according to the aligned phoneme state of the audio feature;

[0034] The determination module is used to determine the lyrics timestamp of the lyrics text according to the phonemes and singing time corresponding to the audio features of each frame.

[0035] In a possible implementation, the identification module is used to:

[0036] Input the audio features of each audio frame in the target song and the phoneme sequence of the lyrics text into the speech recognition model, and obtain the reference phoneme state corresponding to the audio features of each frame output by the speech recognition model in the phoneme sequence and the first probability corresponding to the reference phoneme state;

[0037] Inputting the audio features of each audio frame in the target song and the phoneme set of the language to which the speech recognition model belongs into the speech recognition model, and obtaining a second probability that the audio features of each frame output by the speech recognition model correspond to the reference phoneme state in the phoneme set;

[0038] For each frame of audio features, the confidence that the audio features correspond to the reference phoneme states in the language of the speech recognition model is determined based on a first probability and a second probability that the audio features correspond to the reference phoneme states.

[0039] In a possible implementation, the identification module is used to:

[0040] Input the audio features of each audio frame in the target song and the phoneme sequence of the lyrics text into the speech recognition model, and obtain the probability of the audio features of each frame output by the speech recognition model corresponding to each phoneme state in the phoneme sequence;

[0041] For each of the audio features, the phoneme state corresponding to the maximum probability of the audio feature is used as the reference phoneme state corresponding to the audio feature, and the maximum probability is determined as the first probability that the audio feature corresponds to the reference phoneme state.

[0042] In a possible implementation, the identification module is used to:

[0043] Inputting the audio features of each audio frame in the target song and the phoneme set of the language to which the speech recognition model belongs into the speech recognition model, and obtaining the probability of each phoneme state in the phoneme set corresponding to the audio features of each frame output by the speech recognition model;

[0044] Among the probabilities of the audio feature corresponding to each phoneme state, the probability of the audio feature corresponding to the reference phoneme state is obtained as the second probability.

[0045] In a possible implementation, the identification module is used to:

[0046] The first probability that the audio feature corresponds to the reference phoneme state is divided by the second probability that the audio feature corresponds to the reference phoneme state to obtain the confidence that the audio feature corresponds to the reference phoneme state in the language of the speech recognition model.

[0047] In a possible implementation, the acquisition module is further used to:

[0048] The phoneme sequence of the lyrics text is determined according to a mixed pronunciation dictionary; wherein the mixed pronunciation dictionary is composed of pronunciation dictionaries of the languages ​​to which the multiple speech recognition models belong.

[0049] In a possible implementation, the alignment module is used to:

[0050] According to the correspondence between the phoneme state and the phoneme, the phoneme corresponding to the aligned phoneme state of the audio feature is determined as the phoneme corresponding to the audio feature.

[0051] In a possible implementation, the acquisition module is used to:

[0052] Obtaining the initial lyrics text of the target song;

[0053] The initial lyrics text is processed by filtering out non-lyrics information to obtain the lyrics text of the target song.

[0054] In a third aspect, a computer device is provided, comprising a processor and a memory, wherein the memory stores at least one instruction, and the instruction is loaded and executed by the processor to implement the operations performed by the method for determining lyrics timestamps as described in the first aspect above.

[0055] In a fourth aspect, a computer-readable storage medium is provided, wherein the storage medium stores at least one instruction, and the instruction is loaded and executed by a processor to implement the operations performed by the method for determining lyrics timestamps as described in the first aspect above.

[0056] In a fifth aspect, a computer program product is provided, wherein the storage medium stores at least one instruction, and the instruction is loaded and executed by a processor to implement the operations performed by the method for determining lyrics timestamps as described in the first aspect above.

[0057] The solution shown in the embodiment of the present application includes at least the following beneficial effects:

[0058] In the method for determining the timestamp of lyrics provided in the embodiment of the present application, there is no need to manually mark the timestamp of lyrics, but the timestamp of lyrics is automatically determined by using a model, which can improve the efficiency of determining the timestamp of lyrics. In addition, in the present application, the reference phoneme state corresponding to each frame of audio features in multiple languages ​​and the confidence of the corresponding reference phoneme state are obtained through multiple speech recognition models. Furthermore, for each frame of audio features, among the reference phoneme states corresponding to each other in multiple languages, the reference phoneme state with the highest confidence is selected as the aligned phoneme state of the audio feature. Compared with using a single speech recognition model, the aligned phoneme state selected by the method provided in the present application is more accurate, thereby making the final lyrics timestamp more accurate. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0060] Figure 1 This is a flow chart of a method for determining lyrics timestamps provided in an embodiment of the present application;

[0061] Figure 2 is a flow chart of another method for determining lyrics timestamps provided in an embodiment of the present application;

[0062] Figure 3 is another flow chart of a method for determining lyrics timestamps provided in an embodiment of the present application;

[0063] Figure 4 It is a schematic diagram of the structure of a device for determining lyrics timestamps provided in an embodiment of the present application;

[0064] Figure 5 is a schematic diagram of the structure of a terminal provided in an embodiment of the present application;

[0065] Figure 6 It is a schematic diagram of the structure of the server provided in the embodiment of the present application. DETAILED DESCRIPTION

[0066] In order to make the objectives, technical solutions and advantages of the present application clearer, the implementation methods of the present application will be further described in detail below in conjunction with the accompanying drawings.

[0067] In order to facilitate the understanding of the embodiments of the present application, the terms involved in the embodiments of the present application are explained below:

[0068] 1. Phonemes

[0069] Phonemes are the smallest units of speech divided according to the natural properties of language. One pronunciation action forms one phoneme. For example, the pronunciations of "ah", "bo", and "de" can each correspond to one phoneme. There are many ways to represent phonemes. For example, the phonemes corresponding to the pronunciation of Mandarin can be represented by pinyin. For example, the phonemes corresponding to the pronunciation of the three words "Mandarin" can be represented by "p, u, t, o, ng, h, u, a" respectively. The phonemes corresponding to the pronunciation of English can be represented by phonetic symbols, such as / i / , / I / , / e / , etc.

[0070] 2. Phoneme Status

[0071] The phoneme state is a more detailed speech unit obtained by dividing each phoneme. Generally, one phoneme can correspond to three phoneme states (the phoneme state can be represented by a hidden Markov state). The three phoneme states are: the starting sound state, the continuous sound state and the ending sound state of the phoneme pronunciation.

[0072] 3. Pronunciation Dictionary

[0073] The mapping relationship between characters (words) and phonemes, that is, the phonemes corresponding to each character (word) can be recorded in the pronunciation dictionary. Depending on the language, the pronunciation dictionary can be divided into a Chinese pronunciation dictionary, an English pronunciation dictionary, a Japanese pronunciation dictionary, etc. The mapping relationship between each Chinese character and pinyin can be recorded in the Chinese pronunciation dictionary, and the mapping relationship between each English word and phonetic symbols can be recorded in the English pronunciation dictionary. Among them, pinyin and phonetic symbols are both ways of representing phonemes. Alternatively, the International Phonetic Alphabet (IPA) can be used as a way of representing phonemes in the pronunciation dictionary of each language. Grapheme-to-Phoneme (g2p) can also be used instead of IPA. There is also a pronunciation dictionary, which can be called a mixed pronunciation dictionary, which is a collection of pronunciation dictionaries of various languages.

[0074] An embodiment of the present application provides a method for determining a lyrics timestamp, which can be implemented by a terminal or a server.

[0075] When implemented by a server, a technician can batch-produce lyrics files with lyrics timestamps. In this case, the technician can control the server to execute the method for determining lyrics timestamps of an embodiment of the present application to produce lyrics texts with lyrics timestamps.

[0076] When implemented by a terminal, the terminal may obtain the song audio and lyrics text, and the song audio and lyrics belong to the same song. When the user wants to play the song audio, the lyrics text displays the corresponding lyrics as the song audio is played. In this case, the terminal can determine the timestamp of the lyrics in the lyrics text by executing the method provided in the embodiment of the present application, and then, the lyrics can be highlighted according to the corresponding timestamp. Of course, when implemented by a terminal, the technician may also batch produce lyrics files with lyrics timestamps, and the embodiment of the present application is not limited to the specific application scenario.

[0077] In the method for determining the timestamp of lyrics provided in an embodiment of the present application, the reference phoneme state corresponding to each frame of audio features in multiple languages ​​and the confidence of the corresponding reference phoneme state are obtained through multiple speech recognition models. Furthermore, for each frame of audio features, among the reference phoneme states corresponding to the multiple languages, the reference phoneme state with the highest confidence is selected as the aligned phoneme state of the audio feature. Compared to using a single speech recognition model, the aligned phoneme state selected by the method provided in the present application is more accurate, thereby making the final lyrics timestamp more accurate.

[0078] For ease of understanding, the following Figure 1 , the method for determining the timestamp of lyrics provided in the embodiment of the present application is exemplarily described. Figure 1, input the audio features of the target song into multiple pre-trained speech recognition models, wherein the multiple speech recognition models include speech recognition models of language 1, speech recognition models of language 2, ... speech recognition models of language n, etc. For each frame of audio features, the reference phoneme state 1 corresponding to the audio feature in language 1 and the confidence of the corresponding reference phoneme state 1 can be obtained through the speech recognition model of language 1; the reference phoneme state 2 corresponding to the audio feature in language 2 and the confidence of the corresponding reference phoneme state 2 can be obtained through the speech recognition model of language 2; the reference phoneme state n corresponding to the audio feature in language n and the confidence of the corresponding reference phoneme state n can be obtained through the speech recognition model of language n, and the same is true for other speech recognition models. Then, for the frame of audio features, the reference phoneme state with the highest confidence is selected from the reference phoneme state 1, reference phoneme state 2, ... reference phoneme state n as the aligned phoneme state of the frame of audio features. Furthermore, the lyrics timestamp of the lyrics text is determined according to the aligned phoneme state of each frame of audio features.

[0079] The method for determining the timestamp of lyrics provided in the embodiment of the present application is described below. Figure 2 , the processing of the method may include the following steps:

[0080] Step 201: Get the lyrics text of the target song.

[0081] In practice, the initial lyrics text of the target song can be downloaded from a database or the Internet, or the initial lyrics text of the target song can be obtained locally. Since the initial lyrics text of the song usually includes some non-lyrics information in addition to the lyrics, the initial lyrics text can be filtered out of the non-lyrics information to obtain the lyrics text to be used in subsequent processing.

[0082] Non-lyric information, such as the title of the target song and the corresponding singer, composer, lyricist and other information. Accordingly, the filtering of non-lyric information can include detecting keywords, key symbols, etc., and filtering the initial lyrics text for non-lyric information. For example, "《》" can be detected, and "《》" and the text included in "《》" can be deleted. For another example, sentences including keywords such as "composer", "singer", and "lyricist" can be deleted.

[0083] Step 202: Obtain the phoneme sequence corresponding to the lyrics text.

[0084] The phoneme sequence corresponding to the lyrics text includes the phonemes corresponding to each word in the lyrics text.

[0085] In implementation, the phonemes corresponding to each word in the lyrics text are determined in the mixed pronunciation dictionary. The phonemes corresponding to each word are arranged in the order of the corresponding words in the lyrics text to form a phoneme sequence corresponding to the lyrics text.

[0086] The mixed pronunciation dictionary may include phonemes corresponding to characters (words) in various languages, where the phonemes may be international phonetic symbols. The various languages ​​may be official languages ​​of different countries, such as Chinese, English, Korean, Japanese, etc., or different dialects of a country, such as Cantonese, Fujian, Hakka, Gan, etc.

[0087] Step 203: extract the audio features of the target song.

[0088] In implementation, before extracting audio features, the audio of the target song may be framed, and then audio features may be extracted from each audio frame, with one audio frame corresponding to one audio feature.

[0089] In a possible implementation, the audio feature may be a mel-frequency cepstrum (MFCC) feature.

[0090] Step 204: For each of the multiple pre-trained speech recognition models, the audio features of the target song are input into the speech recognition model to obtain the reference phoneme state corresponding to each frame of audio features in the language to which the speech recognition model belongs and the confidence of the corresponding reference phoneme state.

[0091] Each speech recognition model consists of an acoustic model and a language model. Different speech recognition models are trained using song data samples (including song audio and corresponding lyrics) in different languages. Different language models are composed of the phoneme states corresponding to words in different languages, the self-transition of phoneme states, and the probability of transition between phoneme states.

[0092] For example, multiple speech recognition models correspond to Chinese, English, Korean, and Japanese, respectively. Among them, the speech recognition model corresponding to Chinese is obtained by training Chinese songs and lyrics texts, and the language model in the speech recognition model is composed of the phoneme states corresponding to the characters in the Chinese lyrics, the self-transition of the phoneme states, and the probability of transition between phoneme states; the speech recognition model corresponding to English is obtained by training English songs and lyrics texts, and the language model in the speech recognition model is composed of the phoneme states corresponding to the characters (words) in the English lyrics, the self-transition of the phoneme states, and the probability of transition between phoneme states; the speech recognition model corresponding to Korean is obtained by training Korean songs and lyrics texts, and the language model in the speech recognition model is composed of the phoneme states corresponding to the characters in the Korean lyrics, the self-transition of the phoneme states, and the probability of transition between phoneme states; the speech recognition model corresponding to Japanese is obtained by training Japanese songs and lyrics texts, and the language model in the speech recognition model is composed of the phoneme states corresponding to the characters in the Japanese lyrics, the self-transition of the phoneme states, and the probability of transition between phoneme states.

[0093] For a language model in a speech recognition model of any language, the language model construction method can be as follows:

[0094] A large amount of lyrics text of the language is obtained from the music library, and word segmentation is performed. Then, the corresponding phonemes are added to each word using the International Phonetic Alphabet to obtain the lyrics pronunciation dictionary of the language. Then, all phonemes are separated from the lyrics pronunciation dictionary to construct the phoneme set of the language. Then, three hidden Markov states (phoneme states) are used to represent each phoneme in the phoneme set, and the transition probabilities of self-transition and transition between phoneme states of each phoneme state are counted to obtain the hidden Markov model (also known as the language model) of the language.

[0095] The processing of step 204 is described below. The processing of step 204 may include the following sub-steps:

[0096] Step 2041: For each speech recognition model, the audio features and phoneme sequence of the target song are input into the speech recognition model. For each frame of audio features, the probability of each phoneme state in the phoneme sequence corresponding to the audio feature output by the speech recognition model is obtained, and the phoneme state corresponding to the maximum probability is used as the reference phoneme state corresponding to the audio feature. Accordingly, the maximum probability can be called the first probability of the audio feature corresponding to the reference phoneme state.

[0097] In implementation, the speech recognition model outputs the probability of the audio feature corresponding to each phoneme state according to the decoding graph. When the same audio feature is input, if the decoding graph based on the speech recognition model is different, the probability of the output audio feature corresponding to each phoneme state is also different.

[0098] For each speech recognition model, first, according to the phoneme sequence of the target song and the global decoding graph of the speech recognition model, the decoding graph corresponding to the phoneme sequence is determined, and the decoding graph corresponding to the phoneme sequence is a subset of the global decoding graph of the speech recognition model. Each possible decoding path in the decoding graph corresponding to the phoneme sequence only includes the phoneme states of the common phonemes of the phoneme sequence and the phoneme set of the target language, while the decoding path in the global decoding graph may include the phoneme states of each phoneme in the phoneme set of the target language to which the speech recognition model belongs. Then, the audio sequence of the target song is input into the speech recognition model, so that the speech recognition model outputs the probability of each phoneme state in the phoneme sequence corresponding to each frame of audio features according to the decoding graph corresponding to the phoneme sequence. For each frame of audio features, the phoneme state corresponding to the maximum probability is used as the reference phoneme state corresponding to the audio feature in the language to which the speech recognition belongs, and the corresponding probability is recorded as the first probability.

[0099] For example, Figure 3 As shown, for the speech recognition model of language 1, first, determine the first decoding graph corresponding to the phoneme sequence. Then, input the phoneme sequence into the speech recognition model of language 1. For any frame of audio features of the target song (recorded as audio feature 1), the speech recognition model of language 1 outputs the probability that the audio feature 1 corresponds to phoneme state 1, the probability that it corresponds to phoneme state 2...the probability that it corresponds to phoneme state a, wherein phoneme state 1, phoneme state 2...phoneme state a are the phoneme states of the phoneme sequence and the phoneme set of language 1, that is, the phoneme states in the first decoding graph. Then, among the probabilities of the audio feature 1 corresponding to the various phoneme states in the first decoding graph, the phoneme state corresponding to the maximum probability is selected as the reference phoneme state 1, and accordingly, the maximum probability is used as the first probability of the audio feature 1 corresponding to the reference phoneme state 1. Similarly, for the speech recognition model of language 2, the audio feature 1 corresponding to the reference phoneme state 2 and the first probability corresponding to the reference phoneme state 2 can be obtained. For the speech recognition model of language n, the reference phoneme state n corresponding to audio feature 1 and the first probability corresponding to the reference phoneme state n can be obtained. The specific number of speech recognition models used can be configured according to actual needs, and this application does not limit it.

[0100] Step 2042: For each speech recognition model, the audio features of the target song and the phoneme set of the language to which the speech recognition model belongs are input into the speech recognition model. For each frame of audio features, the probability that the audio feature output by the speech recognition model corresponds to each phoneme state in the phoneme set is obtained. From the probability that the audio feature corresponds to each phoneme state in the phoneme set, a second probability that the audio feature corresponds to a reference phoneme state is obtained.

[0101] In implementation, for each speech recognition model, first, the global decoding graph of the speech recognition model is determined according to the phoneme set of the language to which the speech recognition model belongs. In a possible implementation, the global decoding graph can be preset, and here it is only necessary to obtain the global decoding graph of the speech recognition model locally. Then, the audio sequence of the target song is input into the speech recognition model, so that the speech recognition model outputs the probability of each frame of audio features corresponding to each phoneme state in the phoneme set of the language to which the speech recognition model belongs according to the global decoding graph. For each frame of audio features, the second probability of the audio feature corresponding to its reference phoneme state in the language to which the speech recognition model belongs is obtained from the probability of the audio feature corresponding to each phoneme state in the phoneme set of the language to which the speech recognition model belongs. Among them, the reference phoneme state of the audio feature in the language to which the speech recognition model belongs has been determined in step 2041.

[0102] The processing of step 2042 is illustrated below with reference to the example in step 2041.

[0103] For example, Figure 3 As shown, for the speech recognition model of language 1, the global decoding graph of the speech recognition model is first obtained. Then, the phoneme sequence is input into the speech recognition model of language 1. For any frame of audio features of the target song (recorded as audio feature 1), the speech recognition model of language 1 outputs the probability that the audio feature 1 corresponds to phoneme state 1, the probability that it corresponds to phoneme state 2...the probability that it corresponds to phoneme state b, wherein phoneme state 1, phoneme state 2...phoneme state b are the phoneme states in the phoneme set of language 1, that is, the phoneme states in the global decoding graph. Then, among the probabilities of the audio feature 1 corresponding to the phoneme states in the global decoding graph, the second probability of the reference phoneme state 1 is determined. Similarly, for the speech recognition model of language 2, the second probability of the audio feature 1 corresponding to the reference phoneme state 2 can be obtained. For the speech recognition model of language n, the second probability of the audio feature 1 corresponding to the reference phoneme state n can be obtained.

[0104] Step 2043: For each frame of audio features, divide the first probability of the audio feature corresponding to the reference phoneme state in the language to which the speech recognition model belongs by the second probability to obtain the confidence that the audio feature corresponds to the reference phoneme state in the language to which the speech recognition model belongs.

[0105] The processing of step 2043 is illustrated below with reference to the examples of step 2041 and step 2042.

[0106] For example, Figure 3As shown, for any frame of audio features of the target song (recorded as audio feature 1), the ratio of the first probability of the audio feature 1 corresponding to the reference phoneme state 1 to the second probability of the audio feature 1 corresponding to the reference phoneme state 1 is calculated, and the confidence of the audio feature 1 corresponding to the reference phoneme state 1 in language 1 is obtained. The ratio of the first probability of the audio feature 1 corresponding to the reference phoneme state 2 to the second probability of the audio feature 1 corresponding to the reference phoneme state 2 is calculated, and the confidence of the audio feature 1 corresponding to the reference phoneme state 2 in language 2 is obtained. The ratio of the first probability of the audio feature 1 corresponding to the reference phoneme state n to the second probability of the audio feature 1 corresponding to the reference phoneme state n is calculated, and the confidence of the audio feature 1 corresponding to the reference phoneme state n in language n is obtained.

[0107] Step 205: For each frame of audio features, a reference phoneme state corresponding to a maximum confidence level among multiple confidence levels of the audio feature is used as the aligned phoneme state of the audio feature.

[0108] The processing of step 205 is described below in conjunction with the example of step 2043.

[0109] For example, for any frame of audio features of the target song (referred to as audio feature 1), assume that the confidence of audio feature 1 corresponding to reference phoneme state 1 in language 1 is a1, the confidence of audio feature 1 corresponding to reference phoneme state 2 in language 2 is a2, ..., and the confidence of audio feature 1 corresponding to reference phoneme state n in language n is an. Compare the values ​​of a1, a2, ..., an, and select the reference phoneme state corresponding to the maximum value as the alignment phoneme state corresponding to audio feature 1.

[0110] Step 206: for each frame of audio features, determine the phoneme corresponding to the audio feature according to the aligned phoneme state corresponding to the audio feature and the correspondence between the phoneme state and the phoneme.

[0111] In implementation, each frame of audio features corresponds to an aligned phoneme state. According to the pronunciation time of the audio features, the aligned phoneme states corresponding to the audio features can be formed into an aligned phoneme state sequence. Then, according to the correspondence between the pre-stored phonemes and phoneme states, the phonemes corresponding to each phoneme state in the aligned phoneme state sequence are determined, that is, the phonemes corresponding to each audio frame are determined. These determined phonemes constitute the aligned phoneme sequence corresponding to the aligned phoneme state sequence.

[0112] Step 207: determine the word corresponding to each frame of audio features in the lyrics text according to the phonemes corresponding to each frame of audio features, and determine the lyrics timestamp of the lyrics text according to the singing time corresponding to each frame of audio features.

[0113] The singing time includes the singing start time point and the singing end time point.

[0114] In implementation, a dynamic time warping (DTW) algorithm may be used to determine the recognized lyrics text corresponding to the aligned phoneme sequence determined in step 206. Alternatively, the aligned phoneme sequence may be input into a preset language model to obtain the recognized lyrics text corresponding to the aligned phoneme sequence.

[0115] In this way, since there is a correspondence between the audio features and the phonemes in the aligned phoneme sequence, and there is a correspondence between the phonemes in the aligned phoneme sequence and the characters in the recognized lyrics text, the correspondence between the characters in the recognized lyrics text and the audio features can be determined. Furthermore, based on the singing start time point and the singing end time point of the audio features, the singing start time point and the singing end time point of the characters corresponding to the audio features can be determined. The recognized lyrics text is the lyrics text of the target song, in which the singing start time point and the singing end time point of each character constitute the lyrics timestamp of the lyrics text of the target song.

[0116] In the method for determining the timestamp of lyrics provided in the embodiment of the present application, there is no need to manually mark the timestamp of lyrics, but the timestamp of lyrics is automatically determined by using a model, which can improve the efficiency of determining the timestamp of lyrics. In addition, in the present application, the reference phoneme state corresponding to each frame of audio features in multiple languages ​​and the confidence of the corresponding reference phoneme state are obtained through multiple speech recognition models. Furthermore, for each frame of audio features, among the reference phoneme states corresponding to each other in multiple languages, the reference phoneme state with the highest confidence is selected as the aligned phoneme state of the audio feature. Compared with using a single speech recognition model, the aligned phoneme state selected by the method provided in the present application is more accurate, thereby making the final lyrics timestamp more accurate.

[0117] All the above optional technical solutions can be arbitrarily combined to form optional embodiments of the present disclosure, and will not be described in detail here.

[0118] Based on the same technical concept, the embodiment of the present application also provides a device for determining the timestamp of lyrics, which can be a computer device, such as Figure 4 As shown, the device includes an acquisition module 410, an identification module 420, an alignment module 430 and a determination module 440, wherein:

[0119] The acquisition module 410 is used to acquire the lyrics text of the target song and extract the audio features of each audio frame in the target song;

[0120] The recognition module 420 is used to obtain a plurality of pre-trained speech recognition models, wherein the plurality of speech recognition models are speech recognition models trained with song data samples in different languages ​​respectively; for each of the speech recognition models, the audio features of each audio frame in the target song are input into the speech recognition model to obtain a reference phoneme state corresponding to each frame of the audio features in the language of the speech recognition model and a confidence level of the reference phoneme state;

[0121] an alignment module 430, configured to, for each frame of the audio feature, use a reference phoneme state corresponding to a maximum confidence level among a plurality of confidence levels of the audio feature as an aligned phoneme state of the audio feature, and determine a phoneme corresponding to the audio feature according to the aligned phoneme state of the audio feature;

[0122] The determination module 440 is used to determine the lyrics timestamp of the lyrics text according to the phonemes and singing time corresponding to the audio features of each frame.

[0123] In a possible implementation, the identification module 420 is used to:

[0124] The audio features of each audio frame in the target song and the phoneme sequence of the lyrics text are input into the speech recognition model, and a reference phoneme state corresponding to the audio features of each frame output by the speech recognition model in the phoneme sequence and a first probability corresponding to the reference phoneme state are obtained;

[0125] Inputting the audio features of each audio frame in the target song and the phoneme set of the language to which the speech recognition model belongs into the speech recognition model, and obtaining a second probability that the audio features of each frame output by the speech recognition model correspond to the reference phoneme state in the phoneme set;

[0126] For each frame of audio features, the confidence that the audio features correspond to the reference phoneme states in the language of the speech recognition model is determined based on a first probability and a second probability that the audio features correspond to the reference phoneme states.

[0127] In a possible implementation, the identification module 420 is used to:

[0128] Input the audio features of each audio frame in the target song and the phoneme sequence of the lyrics text into the speech recognition model, and obtain the probability of the audio features of each frame output by the speech recognition model corresponding to each phoneme state in the phoneme sequence;

[0129] For each of the audio features, the phoneme state corresponding to the maximum probability of the audio feature is used as the reference phoneme state corresponding to the audio feature, and the maximum probability is determined as the first probability that the audio feature corresponds to the reference phoneme state.

[0130] In a possible implementation, the identification module 420 is used to:

[0131] Inputting the audio features of each audio frame in the target song and the phoneme set of the language to which the speech recognition model belongs into the speech recognition model, and obtaining the probability of each phoneme state in the phoneme set corresponding to the audio features of each frame output by the speech recognition model;

[0132] Among the probabilities of the audio feature corresponding to each phoneme state, the probability of the audio feature corresponding to the reference phoneme state is obtained as the second probability.

[0133] In a possible implementation, the identification module 420 is used to:

[0134] The first probability that the audio feature corresponds to the reference phoneme state is divided by the second probability that the audio feature corresponds to the reference phoneme state to obtain the confidence that the audio feature corresponds to the reference phoneme state in the language of the speech recognition model.

[0135] In a possible implementation, the acquisition module 410 is further configured to:

[0136] The phoneme sequence of the lyrics text is determined according to a mixed pronunciation dictionary; wherein the mixed pronunciation dictionary is composed of pronunciation dictionaries of the languages ​​to which the multiple speech recognition models belong.

[0137] In a possible implementation, the alignment module 430 is used to:

[0138] According to the correspondence between the phoneme state and the phoneme, the phoneme corresponding to the aligned phoneme state of the audio feature is determined as the phoneme corresponding to the audio feature.

[0139] In a possible implementation, the acquisition module 410 is used to:

[0140] Obtaining the initial lyrics text of the target song;

[0141] The initial lyrics text is processed by filtering out non-lyrics information to obtain the lyrics text of the target song.

[0142] In the method for determining the timestamp of lyrics provided in the embodiment of the present application, there is no need to manually mark the timestamp of lyrics, but the timestamp of lyrics is automatically determined by using a model, which can improve the efficiency of determining the timestamp of lyrics. In addition, in the present application, the reference phoneme state corresponding to each frame of audio features in multiple languages ​​and the confidence of the corresponding reference phoneme state are obtained through multiple speech recognition models. Furthermore, for each frame of audio features, among the reference phoneme states corresponding to each other in multiple languages, the reference phoneme state with the highest confidence is selected as the aligned phoneme state of the audio feature. Compared with using a single speech recognition model, the aligned phoneme state selected by the method provided in the present application is more accurate, thereby making the final lyrics timestamp more accurate.

[0143] It should be noted that: the device for determining lyrics timestamp provided in the above embodiment only uses the division of the above functional modules as an example to illustrate when triggering intelligent network services. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above. In addition, the device for triggering intelligent network services provided in the above embodiment and the method embodiment for triggering intelligent network services belong to the same concept. The specific implementation process is detailed in the method embodiment and will not be repeated here.

[0144] Figure 5 The block diagram of the structure of an electronic device 500 provided by an exemplary embodiment of the present application is shown. The electronic device 500 may be a portable mobile terminal, such as a smart phone, a tablet computer, an MP3 player (Moving Picture Experts Group Audio Layer III), an MP4 player (Moving Picture Experts Group Audio Layer IV), a laptop computer or a desktop computer. The electronic device 500 may also be called a user device, a portable terminal, a laptop terminal, a desktop terminal or other names.

[0145] Typically, the electronic device 500 includes a processor 501 and a memory 502 .

[0146] The processor 501 may include one or more processing cores, such as a 4-core processor, an 8-core processor, etc. The processor 501 may be implemented in at least one hardware form of DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), and PLA (Programmable Logic Array). The processor 501 may also include a main processor and a coprocessor. The main processor is a processor for processing data in the awake state, also known as a CPU (Central Processing Unit); the coprocessor is a low-power processor for processing data in the standby state. In some embodiments, the processor 501 may be integrated with a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content to be displayed on the display screen. In some embodiments, the processor 501 may also include an AI (Artificial Intelligence) processor, which is used to process computing operations related to machine learning.

[0147] The memory 502 may include one or more computer-readable storage media, which may be non-transitory. The memory 502 may also include a high-speed random access memory, and a non-volatile memory, such as one or more disk storage devices, flash memory storage devices. In some embodiments, the non-transitory computer-readable storage medium in the memory 502 is used to store at least one instruction, which is used to be executed by the processor 501 to implement the method for determining the timestamp of lyrics provided in the method embodiment of the present application.

[0148] In some embodiments, the electronic device 500 may further optionally include: a peripheral device interface 503 and at least one peripheral device. The processor 501, the memory 502 and the peripheral device interface 503 may be connected via a bus or a signal line. Each peripheral device may be connected to the peripheral device interface 503 via a bus, a signal line or a circuit board. Specifically, the peripheral device includes: at least one of a radio frequency circuit 504, a display screen 505, a camera assembly 506, an audio circuit 507, a positioning assembly 508 and a power supply 509.

[0149] The peripheral device interface 503 may be used to connect at least one peripheral device related to I / O (Input / Output) to the processor 501 and the memory 502. In some embodiments, the processor 501, the memory 502, and the peripheral device interface 503 are integrated on the same chip or circuit board; in some other embodiments, any one or two of the processor 501, the memory 502, and the peripheral device interface 503 may be implemented on a separate chip or circuit board, which is not limited in this embodiment.

[0150] The radio frequency circuit 504 is used to receive and transmit RF (Radio Frequency) signals, also known as electromagnetic signals. The radio frequency circuit 504 communicates with the communication network and other communication devices through electromagnetic signals. The radio frequency circuit 504 converts electrical signals into electromagnetic signals for transmission, or converts received electromagnetic signals into electrical signals. Optionally, the radio frequency circuit 504 includes: an antenna system, an RF transceiver, one or more amplifiers, a tuner, an oscillator, a digital signal processor, a codec chipset, a user identity module card, etc. The radio frequency circuit 504 can communicate with other terminals through at least one wireless communication protocol. The wireless communication protocol includes but is not limited to: the World Wide Web, a metropolitan area network, an intranet, various generations of mobile communication networks (2G, 3G, 4G and 5G), a wireless local area network and / or a WiFi (Wireless Fidelity) network. In some embodiments, the radio frequency circuit 504 may also include circuits related to NFC (Near Field Communication), which is not limited in this application.

[0151] The display screen 505 is used to display a UI (User Interface). The UI may include graphics, text, icons, videos, and any combination thereof. When the display screen 505 is a touch display screen, the display screen 505 also has the ability to collect touch signals on the surface or above the surface of the display screen 505. The touch signal can be input to the processor 501 as a control signal for processing. At this time, the display screen 505 can also be used to provide virtual buttons and / or virtual keyboards, also known as soft buttons and / or soft keyboards. In some embodiments, the display screen 505 can be one, arranged on the front panel of the electronic device 500; in other embodiments, the display screen 505 can be at least two, respectively arranged on different surfaces of the electronic device 500 or in a folding design; in other embodiments, the display screen 505 can be a flexible display screen, arranged on a curved surface or a folding surface of the electronic device 500. Even, the display screen 505 can also be set to a non-rectangular irregular shape, that is, a special-shaped screen. The display screen 505 can be made of materials such as LCD (Liquid Crystal Display), OLED (Organic Light-Emitting Diode, organic light-emitting diode).

[0152] The camera assembly 506 is used to capture images or videos. Optionally, the camera assembly 506 includes a front camera and a rear camera. Typically, the front camera is arranged on the front panel of the terminal, and the rear camera is arranged on the back of the terminal. In some embodiments, there are at least two rear cameras, which are any one of a main camera, a depth of field camera, a wide-angle camera, and a telephoto camera, so as to realize the fusion of the main camera and the depth of field camera to realize the background blur function, the fusion of the main camera and the wide-angle camera to realize the panoramic shooting and VR (Virtual Reality) shooting function or other fusion shooting functions. In some embodiments, the camera assembly 506 may also include a flash. The flash can be a monochrome temperature flash or a dual-color temperature flash. A dual-color temperature flash refers to a combination of a warm light flash and a cold light flash, which can be used for light compensation at different color temperatures.

[0153] The audio circuit 507 may include a microphone and a speaker. The microphone is used to collect sound waves from the user and the environment, and convert the sound waves into electrical signals and input them into the processor 501 for processing, or input them into the radio frequency circuit 504 to achieve voice communication. For the purpose of stereo acquisition or noise reduction, there may be multiple microphones, which are respectively arranged at different parts of the electronic device 500. The microphone may also be an array microphone or an omnidirectional acquisition microphone. The speaker is used to convert the electrical signal from the processor 501 or the radio frequency circuit 504 into sound waves. The speaker may be a traditional film speaker or a piezoelectric ceramic speaker. When the speaker is a piezoelectric ceramic speaker, it can not only convert the electrical signal into sound waves audible to humans, but also convert the electrical signal into sound waves inaudible to humans for purposes such as ranging. In some embodiments, the audio circuit 507 may also include a headphone jack.

[0154] Positioning component 508 is used to locate the current geographic location of electronic device 500 to implement navigation or LBS (Location Based Service). Positioning component 508 can be a positioning component based on the US GPS (Global Positioning System), China's Beidou system or Russia's Galileo system.

[0155] The power supply 509 is used to power various components in the electronic device 500. The power supply 509 can be an alternating current, a direct current, a disposable battery, or a rechargeable battery. When the power supply 509 includes a rechargeable battery, the rechargeable battery can be a wired rechargeable battery or a wireless rechargeable battery. A wired rechargeable battery is a battery charged through a wired line, and a wireless rechargeable battery is a battery charged through a wireless coil. The rechargeable battery can also be used to support fast charging technology.

[0156] In some embodiments, the electronic device 500 further includes one or more sensors 510 , including but not limited to: an acceleration sensor 511 , a gyroscope sensor 512 , a pressure sensor 513 , a fingerprint sensor 514 , an optical sensor 515 , and a proximity sensor 516 .

[0157] The acceleration sensor 511 can detect the magnitude of acceleration on the three coordinate axes of the coordinate system established by the electronic device 500. For example, the acceleration sensor 511 can be used to detect the components of gravity acceleration on the three coordinate axes. The processor 501 can control the display screen 505 to display the user interface in a horizontal view or a vertical view according to the gravity acceleration signal collected by the acceleration sensor 511. The acceleration sensor 511 can also be used for collecting game or user motion data.

[0158] The gyro sensor 512 can detect the body direction and rotation angle of the electronic device 500, and the gyro sensor 512 can cooperate with the acceleration sensor 511 to collect the user's 3D actions on the electronic device 500. The processor 501 can implement the following functions based on the data collected by the gyro sensor 512: motion sensing (such as changing the UI according to the user's tilt operation), image stabilization during shooting, game control, and inertial navigation.

[0159] The pressure sensor 513 can be set on the side frame of the electronic device 500 and / or the lower layer of the display screen 505. When the pressure sensor 513 is set on the side frame of the electronic device 500, it can detect the user's holding signal of the electronic device 500, and the processor 501 performs left and right hand recognition or shortcut operation according to the holding signal collected by the pressure sensor 513. When the pressure sensor 513 is set on the lower layer of the display screen 505, the processor 501 controls the operability controls on the UI interface according to the user's pressure operation on the display screen 505. The operability controls include at least one of a button control, a scroll bar control, an icon control, and a menu control.

[0160] The fingerprint sensor 514 is used to collect the user's fingerprint, and the processor 501 identifies the user's identity based on the fingerprint collected by the fingerprint sensor 514, or the fingerprint sensor 514 identifies the user's identity based on the collected fingerprint. When the user's identity is identified as a trusted identity, the processor 501 authorizes the user to perform relevant sensitive operations, which include unlocking the screen, viewing encrypted information, downloading software, paying, and changing settings. The fingerprint sensor 514 can be set on the front, back or side of the electronic device 500. When a physical button or a manufacturer logo is set on the electronic device 500, the fingerprint sensor 514 can be integrated with the physical button or the manufacturer logo.

[0161] The optical sensor 515 is used to collect the ambient light intensity. In one embodiment, the processor 501 can control the display brightness of the display screen 505 according to the ambient light intensity collected by the optical sensor 515. Specifically, when the ambient light intensity is high, the display brightness of the display screen 505 is increased; when the ambient light intensity is low, the display brightness of the display screen 505 is reduced. In another embodiment, the processor 501 can also dynamically adjust the shooting parameters of the camera component 506 according to the ambient light intensity collected by the optical sensor 515.

[0162] The proximity sensor 516, also called a distance sensor, is usually disposed on the front panel of the electronic device 500. The proximity sensor 516 is used to collect the distance between the user and the front of the electronic device 500. In one embodiment, when the proximity sensor 516 detects that the distance between the user and the front of the electronic device 500 is gradually decreasing, the processor 501 controls the display screen 505 to switch from the screen-on state to the screen-off state; when the proximity sensor 516 detects that the distance between the user and the front of the electronic device 500 is gradually increasing, the processor 501 controls the display screen 505 to switch from the screen-off state to the screen-on state.

[0163] Those skilled in the art will understand that Figure 5 The structure shown in the figure does not constitute a limitation on the electronic device 500, and may include more or less components than those shown in the figure, or combine some components, or adopt a different component arrangement.

[0164] Figure 6 600 may have relatively large differences due to different configurations or performances, and may include one or more processors (central processing units, CPU) 601 and one or more memories 602, wherein the memory 1002 stores at least one instruction, and the at least one instruction is loaded and executed by the processor 601 to implement the methods provided by the above-mentioned various method embodiments. Of course, the server may also have components such as a wired or wireless network interface, a keyboard, and an input and output interface for input and output, and the server may also include other components for implementing device functions, which will not be described in detail here.

[0165] In an exemplary embodiment, a computer-readable storage medium is also provided, such as a memory including instructions, which can be executed by a processor in a terminal to complete the method for determining the timestamp of lyrics in the above embodiment. The computer-readable storage medium can be non-transitory. For example, the computer-readable storage medium can be a ROM (Read-Only Memory), a RAM (Random Access Memory), a CD-ROM (Compact Disc Read-Only Memory), a magnetic tape, a floppy disk, an optical data storage device, etc.

[0166] It should be noted that the information (including but not limited to user device information, user personal information, etc.), data (including but not limited to data used for analysis, stored data, displayed data, etc.) and signals (including but not limited to signals transmitted between user terminals and other devices, etc.) involved in this application are all authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant laws, regulations and standards of relevant countries and regions. For example, the song audio, lyrics text, etc. involved in this application are all obtained with full authorization.

[0167] A person skilled in the art will understand that all or part of the steps to implement the above embodiments may be accomplished by hardware or by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, and the above-mentioned storage medium may be a read-only memory, a disk or an optical disk, etc.

[0168] The above description is only an optional embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application shall be included in the protection scope of the present application.

Claims

1. A method for determining a lyrics timestamp, characterized in that: The method comprises: Obtaining lyrics text of a target song, and extracting audio features of each audio frame in the target song; Acquire a plurality of pre-trained speech recognition models, wherein the plurality of speech recognition models are speech recognition models trained with song data samples in different languages; For each of the speech recognition models, the audio features of each audio frame in the target song are input into the speech recognition model to obtain the reference phoneme state corresponding to each frame of the audio features in the language of the speech recognition model and the confidence of the reference phoneme state; For each frame of the audio feature, taking a reference phoneme state corresponding to a maximum confidence level among multiple confidence levels of the audio feature as an aligned phoneme state of the audio feature, and determining a phoneme corresponding to the audio feature according to the aligned phoneme state of the audio feature; The lyrics timestamp of the lyrics text is determined according to the phonemes and singing time corresponding to the audio features of each frame.

2. The method according to claim 1, characterized in that The step of inputting the audio features of each audio frame in the target song into the speech recognition model to obtain the reference phoneme state corresponding to each frame of the audio features in the language of the speech recognition model and the confidence of the reference phoneme state includes: Input the audio features of each audio frame in the target song and the phoneme sequence of the lyrics text into the speech recognition model, and obtain the reference phoneme state corresponding to the audio features of each frame output by the speech recognition model in the phoneme sequence and the first probability corresponding to the reference phoneme state; Inputting the audio features of each audio frame in the target song and the phoneme set of the language to which the speech recognition model belongs into the speech recognition model, and obtaining a second probability that the audio features of each frame output by the speech recognition model correspond to the reference phoneme state in the phoneme set; For each frame of audio features, the confidence that the audio features correspond to the reference phoneme states in the language of the speech recognition model is determined based on a first probability and a second probability that the audio features correspond to the reference phoneme states.

3. The method according to claim 2, characterized in that The step of inputting the audio features of each audio frame in the target song and the phoneme sequence of the lyrics text into the speech recognition model, and obtaining the reference phoneme state corresponding to the audio features of each frame output by the speech recognition model in the phoneme sequence and the first probability corresponding to the reference phoneme state includes: Input the audio features of each audio frame in the target song and the phoneme sequence of the lyrics text into the speech recognition model, and obtain the probability of the audio features of each frame output by the speech recognition model corresponding to each phoneme state in the phoneme sequence; For each of the audio features, the phoneme state corresponding to the maximum probability of the audio feature is used as the reference phoneme state corresponding to the audio feature, and the maximum probability is determined as the first probability that the audio feature corresponds to the reference phoneme state.

4. The method according to claim 2, characterized in that: The step of inputting the audio features of each audio frame in the target song and the phoneme set of the language to which the speech recognition model belongs into the speech recognition model, and obtaining a second probability that the audio features of each frame output by the speech recognition model correspond to the reference phoneme state in the phoneme set, comprises: Inputting the audio features of each audio frame in the target song and the phoneme set of the language to which the speech recognition model belongs into the speech recognition model, and obtaining the probability of each phoneme state in the phoneme set corresponding to the audio features of each frame output by the speech recognition model; Among the probabilities that the audio feature corresponds to each phoneme state, the probability that the audio feature corresponds to the reference phoneme state is obtained as the second probability.

5. The method according to claim 2, characterized in that: The step of determining the confidence that the audio feature corresponds to the reference phoneme state in the language of the speech recognition model according to the first probability and the second probability that the audio feature corresponds to the reference phoneme state comprises: The first probability that the audio feature corresponds to the reference phoneme state is divided by the second probability that the audio feature corresponds to the reference phoneme state to obtain the confidence that the audio feature corresponds to the reference phoneme state in the language of the speech recognition model.

6. The method according to claim 2, characterized in that Also includes: The phoneme sequence of the lyrics text is determined according to a mixed pronunciation dictionary; wherein the mixed pronunciation dictionary is composed of pronunciation dictionaries of the languages ​​to which the multiple speech recognition models belong.

7. The method according to any one of claims 1 to 6, characterized in that The determining the phoneme corresponding to the audio feature according to the aligned phoneme state of the audio feature includes: According to the correspondence between the phoneme state and the phoneme, the phoneme corresponding to the aligned phoneme state of the audio feature is determined as the phoneme corresponding to the audio feature.

8. The method according to any one of claims 1 to 6, characterized in that The step of obtaining the lyrics text of the target song includes: Obtaining the initial lyrics text of the target song; The initial lyrics text is processed by filtering out non-lyrics information to obtain the lyrics text of the target song.

9. A computer device, characterized in that: The computer device includes a processor and a memory, wherein at least one instruction is stored in the memory, and the instruction is loaded and executed by the processor to implement the operation performed by the method for determining lyrics timestamps according to any one of claims 1 to 8.

10. A computer-readable storage medium, characterized in that: The storage medium stores at least one instruction, which is loaded and executed by the processor to implement the operation performed by the method for determining lyrics timestamps according to any one of claims 1 to 8.

Images