Feature extraction model training methods, lyric timestamp acquisition methods, and related products

Abstract

This application relates to a feature extraction model training method, a lyrics timestamp acquisition method, a computer device, a readable storage medium, and a program product, belonging to the field of audio processing technology, and can improve the alignment accuracy of song audio and text. The feature extraction model training method includes: acquiring sample song audio and multiple sample lyrics texts corresponding to the sample song audio; determining the sample audio frame corresponding to each sample lyric text from the sample song audio based on the pronunciation stage of each sample lyric text; determining positive sample pairs and negative sample pairs based on the multiple sample audio frames and the sample lyrics texts corresponding to each sample audio frame; and performing comparative learning training on the audio feature extraction module and the text feature extraction module based on the positive and negative sample pairs to obtain a trained feature extraction model; wherein the audio feature extraction module is used to extract the audio features of the audio frames; and the text feature extraction module is used to extract the text features of the lyrics text.

Description

Technical Field

[0001] This application relates to the field of audio processing technology, and in particular to a method for training a feature extraction model, a method for obtaining lyrics timestamps, a computer device, a computer-readable storage medium, and a computer program product. Background Technology

[0002] With the development of computer technology, it is possible to automatically generate lyrics timestamps and quickly align song audio with corresponding text content.

[0003] In related technologies, lyrics timestamps are automatically generated by simultaneously combining acoustic and statistical models. The acoustic model learns the correspondence between states and audio frames, while the statistical model determines the sequential relationships between states. Together, they construct an automatic alignment system for song audio and text. However, practical experience has shown that the alignment accuracy of song audio and text using this method still needs improvement. Summary of the Invention

[0004] Therefore, it is necessary to provide a feature extraction model training method, a lyrics timestamp acquisition method, a computer device, a computer-readable storage medium, and a computer program product to address the aforementioned technical problems.

[0005] Firstly, this application provides a method for obtaining lyrics timestamps, including:

[0006] Obtain sample song audio and multiple sample lyrics texts corresponding to the sample song audio; wherein the sample lyrics texts are obtained by annotating the pronunciation stages of the original lyrics text in the sample lyrics text corresponding to the sample song audio.

[0007] Based on the pronunciation stage of each sample lyric, determine the sample audio frame corresponding to each sample lyric from the sample song audio;

[0008] Based on the multiple sample audio frames and the sample lyrics text corresponding to each sample audio frame, positive sample pairs and negative sample pairs are determined; the positive sample pair includes the sample audio frame and the sample lyrics text corresponding to the sample audio frame, and the negative sample pair includes the sample audio frame and the sample lyrics text that does not correspond to the sample audio frame.

[0009] Based on the positive and negative sample pairs, the audio feature extraction module and the text feature extraction module are trained through comparative learning to obtain a trained feature extraction model; wherein the audio feature extraction module is used to extract audio features of audio frames; and the text feature extraction module is used to extract text features of lyrics.

[0010] In one embodiment, the step of performing comparative learning training on the audio feature extraction module and the text feature extraction module based on the positive sample pairs and the negative sample pairs to obtain a trained feature extraction model includes:

[0011] The audio feature extraction module obtains the audio features of each of the multiple sample audio frames, and the text feature extraction module determines the text features of each of the sample lyrics.

[0012] The contrast loss value is determined based on the first feature difference and the second feature difference; the first feature difference is the difference between the audio features of the sample audio frame and the text features of the sample lyrics in the positive sample pair, and the second feature difference is the difference between the audio features of the sample audio frame and the text features of the sample lyrics in the negative sample pair.

[0013] The module parameters of the audio feature extraction module and the text feature extraction module are adjusted based on the contrast loss value to obtain the trained feature extraction model.

[0014] In one embodiment, determining the sample audio frame corresponding to each sample lyric character from the sample song audio based on the pronunciation stage of each sample lyric character includes:

[0015] Based on the timestamp of the lyrics corresponding to the sample lyrics text, determine multiple audio frames associated with each original lyric text from the sample song audio.

[0016] For each sample lyric text, the multiple audio frames associated with the original lyric text corresponding to the sample lyric text are taken as the multiple audio frames associated with the sample lyric text.

[0017] For each of the sample lyrics, the sample audio frame corresponding to the sample lyrics is determined from the sample song audio based on the pronunciation stage of the sample lyrics and the pronunciation stage of each audio frame in the plurality of audio frames associated with the sample lyrics.

[0018] In one embodiment, the pronunciation stage is a pronunciation start stage, a pronunciation middle stage, or a pronunciation end stage; determining the sample audio frame corresponding to the sample lyrics from the sample song audio based on the pronunciation stage of the sample lyrics and the pronunciation stage of each audio frame in the plurality of audio frames associated with the sample lyrics includes:

[0019] If the pronunciation stage of the sample lyrics is the middle stage of pronunciation, the audio frames whose pronunciation stage is marked as the middle stage of pronunciation among the multiple audio frames associated with the sample lyrics are determined as the sample audio frames corresponding to the sample lyrics.

[0020] If the pronunciation stage of the sample lyrics is the pronunciation start stage, the audio frame whose pronunciation stage is marked as the pronunciation start stage among the multiple audio frames associated with the sample lyrics is determined as the sample audio frame corresponding to the sample lyrics.

[0021] If the pronunciation stage of the sample lyrics is the end of pronunciation, the audio frame whose pronunciation stage is marked as the end of pronunciation among the multiple audio frames associated with the sample lyrics is determined as the sample audio frame corresponding to the sample lyrics.

[0022] In one embodiment, the pronunciation stage is a pronunciation start stage, a pronunciation middle stage, or a pronunciation end stage; determining the sample audio frame corresponding to the sample lyrics from the sample song audio based on the pronunciation stage of the sample lyrics and the pronunciation stage of each audio frame in the plurality of audio frames associated with the sample lyrics includes:

[0023] If the pronunciation stage of the sample lyrics is the middle stage of pronunciation, the audio frame marked as the middle stage of pronunciation among the multiple audio frames associated with the sample lyrics is determined as the sample audio frame corresponding to the sample lyrics.

[0024] If the pronunciation stage of the sample lyrics is the pronunciation start stage, the audio frames marked as the pronunciation start stage in the plurality of audio frames associated with the sample lyrics, and the audio frames before the pronunciation start stage in the sample song audio, a preset number of audio frames are determined as the sample audio frames corresponding to the sample lyrics.

[0025] If the pronunciation stage of the sample lyrics is the end of pronunciation, the audio frames marked as the end of pronunciation stage in the plurality of audio frames associated with the sample lyrics, and a preset number of audio frames after the audio frames of the end of pronunciation stage in the sample song audio, are determined as the sample audio frames corresponding to the sample lyrics.

[0026] In one embodiment, the pronunciation phase is a pronunciation initiation phase, a pronunciation middle phase, or a pronunciation end phase; the method further includes:

[0027] Obtain the audio frame energy of each audio frame in the multiple audio frames associated with the original lyrics text;

[0028] The audio frames whose energy is greater than or equal to the energy threshold and are in the middle of the plurality of audio frames are marked as audio frames in the middle stage of pronunciation.

[0029] Audio frames whose energy is less than the energy threshold and which are in the middle stage of pronunciation among the multiple audio frames are marked as audio frames in the beginning stage of pronunciation.

[0030] Audio frames whose energy is less than the energy threshold and which are located in the middle stage of pronunciation among the plurality of audio frames are marked as audio frames in the end stage of pronunciation.

[0031] Secondly, this application also provides a method for obtaining lyrics timestamps, the method comprising:

[0032] Obtain the song audio and the corresponding lyrics text, wherein the song audio includes multiple song audio frames and the lyrics text includes multiple lyrics text;

[0033] The song audio is input into a trained feature extraction model, and the audio feature extraction module in the feature extraction model performs feature extraction on at least some of the song audio frames in the plurality of song audio frames to obtain the lyrics text features corresponding to each of the at least some of the song audio frames. The feature extraction model is trained by the feature extraction model training method described above.

[0034] The text features of each lyric text in the plurality of lyric texts are obtained from the pre-constructed feature dictionary; the text features of the lyric texts in the feature dictionary are extracted by the text feature extraction module in the feature extraction model.

[0035] Based on the similarity between the text features of the lyrics corresponding to each audio frame of the song and the text features of the lyrics corresponding to each lyric text, the lyrics text corresponding to each audio frame of the song is determined;

[0036] Based on the lyrics text corresponding to each audio frame of the song, obtain the lyrics timestamp of the song audio.

[0037] Thirdly, this application also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of the feature extraction model training method or the lyrics timestamp acquisition method described above.

[0038] Fourthly, this application also provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of the feature extraction model training method or the steps of the lyrics timestamp acquisition method described above.

[0039] Fifthly, this application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the feature extraction model training method or the lyrics timestamp acquisition method described above.

[0040] The aforementioned feature extraction model training method determines the sample audio frame corresponding to each sample lyric text from the sample song audio based on the pronunciation stage of each sample lyric text. Based on multiple sample audio frames and the sample lyric text corresponding to each sample audio frame, positive and negative sample pairs are determined. A positive sample pair includes the sample audio frame and the corresponding sample lyric text, while a negative sample pair includes the sample audio frame and sample lyric text that does not correspond to the sample audio frame. Then, based on the positive and negative sample pairs, the audio feature extraction module and the text feature extraction module are trained through comparative learning to obtain the feature extraction model. This training method enables the trained feature extraction model to learn the mapping relationship between the audio feature space and the text feature space. This mapping relationship, when applied to lyric text alignment scenarios, can more accurately align the song audio and the song text. Attached Figure Description

[0041] To more clearly illustrate the technical solutions in the embodiments of this application or related technologies, the drawings used in the description of the embodiments of this application or related technologies will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0042] Figure 1 This is a flowchart illustrating a feature extraction model training method in one embodiment;

[0043] Figure 2 This is a flowchart illustrating another method for training a feature extraction model in one embodiment;

[0044] Figure 3 This is a flowchart illustrating one step in determining the sample audio frame corresponding to sample lyrics text in one embodiment.

[0045] Figure 4 This is a flowchart illustrating a method for obtaining lyrics timestamps in one embodiment;

[0046] Figure 5 This is an internal structural diagram of a computer device in one embodiment;

[0047] Figure 6 This is an internal structural diagram of another computer device in one embodiment. Detailed Implementation

[0048] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0049] To enable those skilled in the art to better understand this application, the relevant technologies are described below.

[0050] With the development of computer technology, it is now possible to quickly align song audio with corresponding text content by automatically generating lyric timestamps. This automatic lyric timestamp generation involves using an automatic alignment algorithm to determine the start and end times of each syllable in the song audio and its corresponding text. Through automatic lyric alignment, the manual cost of timestamp annotation will be significantly reduced, and the production process for musicians will be simplified.

[0051] In related technologies, lyrics timestamps are mainly generated automatically by combining acoustic and statistical models simultaneously. In one example, the lyrics timestamp model adopts a modeling scheme based on DNN (Deep Neural Network)-HMM (Hidden Markov Model). In this scheme, the DNN model is responsible for learning the correspondence between states and frames, while the HMM model is responsible for the correlation between states. The two are combined to construct a non-end-to-end automatic alignment system.

[0052] Specifically, in this alignment system, on the one hand, spectral features are first extracted from the song audio and input into a DNN model. The classification layer (such as the Softmax layer) of the DNN model outputs the predicted probabilities of each category, and the number of categories output is consistent with the number of states in the state space. On the other hand, the system converts the lyrics text into a phoneme sequence. For example, a text-to-phoneme conversion dictionary can be obtained in advance, and then the lyrics text can be converted into a phoneme sequence containing multiple phonemes using this dictionary. For Chinese, Pinyin can be used, and for English, the International Phonetic Alphabet can be used. After obtaining the phoneme sequence, the phoneme sequence can be converted into a state sequence of an Hidden Markov Model (HMM). Generally, one phoneme corresponds to several states (such as three states). These states can transition from one state to another, and the specific state transition probabilities are described by the Hidden Markov Model. In this example, a state can be understood as the smallest granularity of a word, with one frame of audio features corresponding to one state.

[0053] Then, for the probability of each frame output previously belonging to each state in the state space, the HMM outputs the state transition probability, obtains the path with the maximum global probability through Viterbi decoding, aligns the frames and states one by one to obtain the decoding result (that is, the state corresponding to each frame), and then converts the state corresponding to each frame into phonemes and then converts the phonemes into words to obtain the lyrics timestamp.

[0054] However, it is found in practice that the alignment accuracy of the song audio and the text in the above manner is relatively low, and there is still much room for improvement. Based on this, the present application provides a method for training a feature extraction model, a method for obtaining lyrics timestamps, a computer device, a readable storage medium, and a program product to at least solve the above technical problems.

[0055] In one embodiment, as Figure 1 shown, a method for training a feature extraction model is provided. In this embodiment, an example is given where this method is applied to a server. It can be understood that this method can also be applied to a terminal, and can also be applied to a system including a terminal and a server, and is implemented through the interaction between the terminal and the server. In this embodiment, the method for training a feature extraction model includes the following steps S101 to S104:

[0056] S101, obtaining a sample song audio and a plurality of sample lyrics corresponding to the sample song audio; wherein the sample lyrics are obtained by annotating the pronunciation stage of the original lyrics in the sample lyric text corresponding to the sample song audio.

[0057] In specific implementation, the song audio for training the feature extraction model can be obtained. For the convenience of distinction, the song audio for training the model is called the sample song audio; the sample song audio has a corresponding sample lyric text, and the sample lyric text includes one or more lyric words, and the lyric word is the original lyric word in the sample lyrics. To distinguish it from other lyric words, this lyric word is called the original lyric word.

[0058] During the singing process, the acoustic features of the same original lyric word may be different at different pronunciation stages. For example, due to the change of the singer's breath or the use of singing trill techniques, the acoustic features shown in different stages of the beginning stage, the middle stage, and the end stage of the same original lyric word during the singing process will be different. In this regard, in this embodiment, the pronunciation stage of the original lyric word can be pre-annotated to obtain the corresponding sample lyrics. Different from the original lyric word without the pronunciation stage, the sample lyrics are the lyric words annotated with the pronunciation stage. For example, for the original lyric word "I", after annotating the pronunciation stage of this word, three sample lyrics can be obtained, namely "I_start", "I_middle", and "I_end".

[0059] S102, based on the pronunciation stage of each sample lyric, determine the sample audio frame corresponding to each sample lyric from the sample song audio.

[0060] After obtaining multiple sample lyrics, the audio of the sample song can be analyzed to identify audio frames that are related to the sample lyrics and match the pronunciation stage. These audio frames are also called the sample audio frames corresponding to the sample lyrics. For example, an audio frame containing the acoustic signal corresponding to the original lyrics and whose pronunciation stage matches the pronunciation stage marked in the sample lyrics can be used as the sample audio frame corresponding to the sample lyrics.

[0061] In some examples, for each original lyric text corresponding to a sample lyric text (that is, the sample lyric text obtained by annotating the pronunciation stages of the original lyric text), the audio frames of the original lyric text within the corresponding time range can be determined based on the start and end times of the lyric text indicated by the word timestamps in the sample lyric text. Then, the sample audio frames corresponding to the sample lyric text can be determined from the audio frames within that time range or from audio frames adjacent to that time range.

[0062] S103, based on multiple sample audio frames and the sample lyrics text corresponding to each sample audio frame, determine positive sample pairs and negative sample pairs; positive sample pairs include sample audio frames and the sample lyrics text corresponding to the sample audio frames, and negative sample pairs include sample audio frames and sample lyrics text that do not correspond to the sample audio frames.

[0063] After determining the sample audio frame corresponding to each sample lyric text, the sample lyric text corresponding to each sample audio frame can be obtained based on the correspondence between the sample lyric text and the sample audio frame. Then, based on the sample lyric text corresponding to each sample audio frame, positive sample pairs and negative sample pairs can be constructed. In other words, in this embodiment, pairs are formed on a single sample audio frame basis. A sample pair may include one sample audio frame and one sample lyric text. A positive sample pair, also called a positive example, refers to a training sample constructed from a sample audio frame and the sample lyric text corresponding to that sample audio frame. A negative sample pair, also called a negative example, refers to a training sample constructed from a sample audio and sample lyric text that does not correspond to that sample audio frame. The sample lyric text that does not correspond to the sample audio frame is any lyric text other than the sample lyric text corresponding to that sample audio frame.

[0064] by Figure 2For example, a sample lyric text "我_起始" and a sample audio frame corresponding to "我_起始" can form a positive sample pair. Any sample audio frame corresponding to the sample lyric text "我_起始" and "爱_起始", or any sample audio frame corresponding to the sample lyric text "我_起始" and "我_结束" can form a negative sample pair. That is, for each sample audio frame, the sample audio frame and its corresponding sample lyric text form a positive example, and the sample audio frame and any other non-corresponding sample lyric text form one or more negative examples.

[0065] S104. According to the positive sample pairs and negative sample pairs, perform contrastive learning training on the audio feature extraction module and the text feature extraction module to obtain a trained feature extraction model; wherein the audio feature extraction module is used to extract the audio features of audio frames; the text feature extraction module is used to extract the text features of lyric texts.

[0066] After constructing multiple positive sample pairs and multiple negative sample pairs, the multiple positive sample pairs and multiple negative sample pairs can be used to perform self-supervised contrastive learning training on the audio feature extraction module and the text feature extraction module, so that for audio and text with the same or similar semantics, the feature extraction results output by the two modules can be as close as possible, while for audio and text with different or irrelevant semantics, the feature extraction results output by the two modules can be as different as possible. Thus, when the training end condition is satisfied, for audio and text representing the same semantics, the features output by the two modules are the same or relatively similar. Therefore, the features in different feature spaces can be transformed and mapped to obtain the mapping relationship between the text feature space and the audio feature space.

[0067] In the trained feature extraction model, the mapping relationship between the audio feature space corresponding to the audio feature extraction module and the text feature space corresponding to the text feature extraction module has been constructed. Subsequently, the audio features of audio frames and the text features of words can be extracted using the feature extraction model. Since the mapping relationship between the audio features and the text features has been determined, the corresponding text features can be quickly determined according to the audio features, and thus the audio frames and the corresponding lyric texts can be directly aligned at the character dimension.

[0068] In some alternative embodiments, the audio feature extraction module and the text feature extraction module can be implemented using large models. For example, for audio, the audio feature extraction module can be constructed using a large model such as MERT (Music Understanding Model with Large-Scale Self-supervised Training) or other large models; for text, the text feature extraction module can be constructed using a large model such as BERT (Bidirectional Encoder Representations from Transformers) or other large models. During the contrastive learning training process, the training objective may include: for related audio and text, minimizing the difference between the features output by the audio feature extraction module and the text feature extraction module; for unrelated audio and text, maximizing the difference between the features output by the audio feature extraction module and the text feature extraction module.

[0069] In this embodiment, based on the pronunciation stage of each sample lyric, a sample audio frame corresponding to each sample lyric is determined from the sample song audio. Based on multiple sample audio frames and the sample lyric corresponding to each sample audio frame, positive and negative sample pairs are determined. A positive sample pair includes the sample audio frame and the corresponding sample lyric, while a negative sample pair includes the sample audio frame and sample lyric that does not correspond to the sample audio frame. Then, based on the positive and negative sample pairs, the audio feature extraction module and the text feature extraction module are trained through comparative learning to obtain a feature extraction model. This training method enables the trained feature extraction model to learn the mapping relationship between the audio feature space and the text feature space. This mapping relationship, when applied to lyric alignment scenarios, can more accurately align the song audio with the song text.

[0070] In one embodiment, step S104, which involves comparative learning training of the audio feature extraction module and the text feature extraction module based on positive sample pairs and negative sample pairs to obtain a trained feature extraction model, may include the following steps:

[0071] S1031, the audio feature extraction module obtains the audio features of each sample audio frame in multiple sample audio frames, and the text feature extraction module determines the text features of each sample lyric text.

[0072] Specifically, each sample audio frame and each sample lyric text can be input into the audio feature extraction module and the text feature extraction module, respectively, to obtain the audio features of each sample audio frame and the text features of each sample lyric text.

[0073] S1032, determine the contrast loss value based on the first feature difference and the second feature difference; the first feature difference is the difference between the audio features of the sample audio frame in the positive sample pair and the text features of the sample lyrics, and the second feature difference is the difference between the audio features of the sample audio frame in the negative sample pair and the text features of the sample lyrics.

[0074] After obtaining the audio features corresponding to multiple sample audio frames and the text features corresponding to multiple sample lyrics, the contrastive loss value during the contrastive learning training process is determined on a positive or negative sample pair basis. Specifically, on the one hand, for each positive sample pair, the difference between the audio features of the sample audio frames and the text features of the sample lyrics can be determined; for ease of distinction, this feature difference is called the first feature difference. On the other hand, for each negative sample pair, the difference between the audio features of the sample audio frames and the features of the non-corresponding sample lyrics can be determined; for ease of distinction, this feature difference is called the second feature difference. Since the training objective of contrastive learning is to make the features corresponding to audio and text with the same or similar semantics as close as possible, while making the feature extraction results of audio and text with different or unrelated semantics as different as possible, the first and second feature differences can reflect the training status of the audio feature extraction module and the text feature extraction module respectively, and the corresponding contrastive loss value can be calculated.

[0075] S1033, adjust the module parameters of the audio feature extraction module and the text feature extraction module according to the contrast loss value to obtain the trained feature extraction model.

[0076] In some embodiments, after obtaining the contrastive loss value, the backpropagation algorithm can be used to adjust the module parameters of the audio feature extraction module and the text feature extraction module according to the contrastive loss value, and then return to step S101 to repeat the iteration multiple times until the training termination condition is met, and the trained feature extraction model is obtained.

[0077] In this embodiment, the contrast loss value is determined based on the first feature difference and the second feature difference. Then, the corresponding module parameters of the audio feature extraction module and the text feature extraction module are adjusted. Through self-supervised learning, the audio feature extraction module and the text feature extraction module can efficiently learn the feature representations associated with audio and text as well as the feature representations unrelated to audio and text. No additional training labels are required, and the mapping relationship between the audio feature space and the text feature space can be quickly constructed.

[0078] In one embodiment, step S1032, determining the contrast loss value based on the first feature difference and the second feature difference, may include the following steps: determining a first loss value based on the first feature difference; determining a second loss value based on the second feature difference; and determining a contrast loss value based on the first loss value and the second loss value. Wherein, the first loss value is positively correlated with the first feature difference; and the second loss value is negatively correlated with the second feature difference.

[0079] In practical applications, the first loss value is determined based on the difference in the first feature. This first loss value is positively correlated with the difference in the first feature; that is, the smaller the difference between the audio features of the sample audio frame and the text features of the sample lyrics in a positive sample pair, the smaller the first loss value. Therefore, for associated audio and text, this can guide the audio feature extraction module and the text feature extraction module to output more similar feature representations. On the other hand, the second loss value is determined based on the difference in the second feature; this second loss value is negatively correlated with the difference in the second feature. That is, the greater the difference between the audio features of the sample audio frame and the text features of the non-corresponding sample lyrics in a negative sample pair, the smaller the second loss value. Therefore, for unrelated audio and text, this can guide the audio feature extraction module and the text feature extraction module to output more distinct feature representations, increasing the discriminative power of the output features for text or audio with different semantics.

[0080] The contrastive loss value can then be determined by combining the first loss value and the second loss value. In an exemplary embodiment, the contrastive loss value Lq can be determined by the following formula:

[0081]

[0082] Here, the numerator is the dot product of q and k+, representing the distance between vectors q and k+. In this example, vectors q and k+ are the audio and text features in the positive sample pair, respectively. The denominator is the sum of the dot products of all positive and negative examples. The parameter τ can be used to control the range of the loss function. In one example, q can be understood as the output of the audio feature extraction module in the fully connected layer, which can be represented by embedding. k+ is the output of the corresponding sample lyrics text in the fully connected layer of the text feature extraction module, which can also be represented by embedding.

[0083] In one embodiment, such as Figure 3 As shown, in step S102, determining the sample audio frame corresponding to each sample lyric text from the sample song audio based on the pronunciation stage of each sample lyric text can include the following steps:

[0084] S301, Based on the timestamp of the lyrics corresponding to the sample lyrics text, determine multiple audio frames associated with each original lyric text from the audio of the sample song.

[0085] In practical applications, the timestamps corresponding to the sample lyrics text can be obtained in advance. In some embodiments, the original lyrics text can be determined using existing lyrics timestamp models, such as... Figure 2 As shown, the sample song audio and the corresponding sample lyrics text can be input into an existing DNN-HMM model to obtain the lyrics timestamp output by the DNN-HMM model. The lyrics timestamp includes the word timestamp of each original lyric text in the sample lyrics text. The word timestamp can record the start and end times of the audio signal containing the original lyrics text in the sample song audio.

[0086] Then, based on the lyrics timestamps, multiple audio frames associated with each original lyric can be determined from the sample song audio, such as... Figure 2 As shown, the six audio frames associated with each of the three original lyrics “I”, “love”, and “you” can be determined based on the lyrics timestamps.

[0087] S302, for each sample lyric text, the multiple audio frames associated with the original lyric text corresponding to the sample lyric text are taken as the multiple audio frames associated with the sample lyric text.

[0088] For each original lyric text in the sample lyrics text, by annotating each original lyric text with different pronunciation stages, multiple sample lyric texts corresponding to the original lyric text can be obtained. For each sample lyric text obtained after annotation, the multiple audio frames associated with the original lyric text corresponding to that sample lyric text are used as the multiple audio frames associated with that sample lyric text, which helps to quickly locate the sample audio frames corresponding to the sample lyric text.

[0089] S303, for each sample lyric text, determine the sample audio frame corresponding to the sample lyric text from the sample song audio based on the pronunciation stage of the sample lyric text and the pronunciation stage of each audio frame in the multiple audio frames associated with the sample lyric text.

[0090] In practical applications, pronunciation stages can be identified across multiple audio frames associated with sample lyrics. For example... Figure 2 In the three sample lyrics “I_start”, “I_middle” and “I_end”, the multiple audio frames associated with each sample lyrics are the same as the 6 audio frames associated with the original lyrics “I”. The pronunciation stage can be identified in these 6 audio frames to obtain the pronunciation stage of each audio frame in the multiple audio frames associated with the sample lyrics.

[0091] In some exemplary embodiments, the pronunciation stage can be identified based on the audio frame energy, or it can be divided into multiple pronunciation stages based on the temporal sequence corresponding to the audio frame. For example, if the associated multiple audio frames are divided equally according to the number and order of the pronunciation stages, the 6 audio frames associated with the sample lyrics can be divided into 3 pronunciation stages, and each pronunciation stage includes 2 audio frames.

[0092] Furthermore, by combining the pronunciation stages of the sample lyrics text and the pronunciation stages of each audio frame in the multiple audio frames associated with the sample lyrics text, the sample audio frame corresponding to the sample lyrics text is determined. Specifically, audio frames in the same pronunciation stage as the sample lyrics text can be used as the sample audio frame corresponding to the sample lyrics text; alternatively, the sample audio frame corresponding to the sample lyrics text can be obtained by expanding upon the audio frames in the same pronunciation stage as the sample lyrics text, and then using the expansion result.

[0093] In this embodiment, the sample audio frame corresponding to the sample lyrics can be quickly and accurately matched according to the pronunciation stage of the sample lyrics and the pronunciation stage of each audio frame in the multiple audio frames associated with the sample lyrics. Model training is performed through this correspondence, which helps the model to accurately extract audio features and text features under different pronunciation stages, thereby obtaining a reliable mapping relationship between audio features and text features under different pronunciation stages.

[0094] In one embodiment, the pronunciation stage is a pronunciation start stage, a pronunciation middle stage, or a pronunciation end stage; step S303, determining the sample audio frame corresponding to the sample lyrics text from the sample song audio based on the pronunciation stage of the sample lyrics text and the pronunciation stage of each audio frame in the multiple audio frames associated with the sample lyrics text, may include the following steps:

[0095] If the pronunciation stage of the sample lyrics is the middle stage, the audio frames whose pronunciation stage is marked as the middle stage are identified as the sample audio frames corresponding to the sample lyrics. If the pronunciation stage of the sample lyrics is the beginning stage, the audio frames whose pronunciation stage is marked as the beginning stage are identified as the sample audio frames corresponding to the sample lyrics. If the pronunciation stage of the sample lyrics is the end stage, the audio frames whose pronunciation stage is marked as the end stage are identified as the sample audio frames corresponding to the sample lyrics.

[0096] Specifically, after determining the pronunciation stage marked for each of the multiple audio frames associated with the sample lyric text, the audio frames among the multiple audio frames associated with the sample lyric text that have the same pronunciation stage as the sample lyric text can be determined as the sample audio frames corresponding to the sample lyric text. Based on this, if the pronunciation stage of the sample lyric text is the intermediate pronunciation stage, the audio frames among the multiple audio frames associated with the sample lyric text that are marked with the intermediate pronunciation stage are determined as the sample audio frames corresponding to the sample lyric text; if the pronunciation stage of the sample lyric text is the starting pronunciation stage, the audio frames among the multiple audio frames associated with the sample lyric text that are marked with the starting pronunciation stage are determined as the sample audio frames corresponding to the sample lyric text; if the pronunciation stage of the sample lyric text is the ending pronunciation stage, the audio frames among the multiple audio frames associated with the sample lyric text that are marked with the ending pronunciation stage are determined as the sample audio frames corresponding to the sample lyric text.

[0097] Continuing with Figure 2 as an example for illustration, according to the pronunciation stage recognition results of the 6 audio frames associated with the sample lyric text "I_start", the audio frames in the starting pronunciation stage are the first two audio frames (such as Figure 2 the two audio frames boxed by the dashed line), and these two audio frames can be used as the sample audio frames corresponding to the sample lyric text "I_start".

[0098] In this embodiment, the sample audio frames corresponding to the sample lyric text are directly determined from the multiple audio frames associated with the sample lyric text, which can save the recognition time of the sample audio frames corresponding to the sample lyric text.

[0099] In another embodiment, the pronunciation stage is the starting pronunciation stage, the intermediate pronunciation stage or the ending pronunciation stage; in step S203, according to the pronunciation stage of the sample lyric text and the pronunciation stage of each of the multiple audio frames associated with the sample lyric text, determining the sample audio frames corresponding to the sample lyric text from the sample song audio may include the following steps:

[0100] If the pronunciation stage of the sample lyric text is the intermediate pronunciation stage, the audio frames among the multiple audio frames associated with the sample lyric text that are marked with the intermediate pronunciation stage are determined as the sample audio frames corresponding to the sample lyric text; if the pronunciation stage of the sample lyric text is the starting pronunciation stage, the audio frames among the multiple audio frames associated with the sample lyric text that are marked with the starting pronunciation stage, and a preset number of audio frames before the audio frames in the starting pronunciation stage in the sample song audio are determined as the sample audio frames corresponding to the sample lyric text; if the pronunciation stage of the sample lyric text is the ending pronunciation stage, the audio frames among the multiple audio frames associated with the sample lyric text that are marked with the ending pronunciation stage, and a preset number of audio frames after the audio frames in the ending pronunciation stage in the sample song audio are determined as the sample audio frames corresponding to the sample lyric text.

[0101] Specifically, for the sample lyric text in the middle stage of pronunciation, the audio frames marked as the middle stage of pronunciation among the multiple audio frames associated with the sample lyric text can be directly determined as the sample audio frames corresponding to the sample lyric text.

[0102] For the audio frames marked with the start stage of pronunciation and the pronunciation stage, since the range corresponding to the audio frame is actually determined by the lyric timestamp, due to various factors, the lyric timestamp may be offset, resulting in the start time of the lyric timestamp may not truly cover the actual start time of the original lyric text, thus causing the content in the audio frame corresponding to the actual start time to be misidentified as ambient sound. Similarly, the end time of the lyric timestamp may not truly cover the actual end time of the original lyric text. For example, the end time indicated by the lyric timestamp is earlier than the actual end time, thus causing the content in the audio frame corresponding to the actual end time to be misidentified as ambient sound.

[0103] In this regard, in this embodiment, if the pronunciation stage of the sample lyric text is the start stage of pronunciation, in addition to determining the audio frames marked as the start stage of pronunciation among the multiple audio frames associated with the sample lyric text as the sample audio frames corresponding to the sample lyric text, a preset number of audio frames before the audio frames in the start stage of pronunciation in the sample song audio can also be determined as the sample audio frames corresponding to the sample lyric text. If the pronunciation stage of the sample lyric text is the end stage of pronunciation, in addition to determining the audio frames marked as the end stage of pronunciation among the multiple audio frames associated with the sample lyric text as the sample audio frames corresponding to the sample lyric text, a preset number of audio frames after the audio frames in the end stage of pronunciation in the sample song audio can also be determined as the sample audio frames corresponding to the sample lyric text.

[0104] For example, as Figure 2 shown, for the 2 audio frames in the start stage of pronunciation associated with the sample lyric text "I_start", 1 audio frame can be expanded forward to obtain 3 sample audio frames corresponding to the sample lyric text "I_start", that is Figure 2 the 3 audio frames boxed by the solid line frame in

[0105] In this embodiment, by expanding the audio frames in the start stage of pronunciation and the end stage of pronunciation, the audio frames that may be misidentified as ambient sound but are actually related to the lyric text can be correctly recognized, avoiding omission, thereby improving the accuracy of the model in recognizing the audio frame features and text features in different pronunciation states.

[0106] In one embodiment, the pronunciation stage is the start stage of pronunciation, the middle stage of pronunciation or the end stage of pronunciation; the method may further include the following steps:

[0107] Obtain the audio frame energy of each audio frame in multiple audio frames associated with the original lyrics text; label the audio frame whose audio frame energy is greater than or equal to the energy threshold and is in the middle of the multiple audio frames as the audio frame in the middle stage of pronunciation; label the audio frame whose audio frame energy is less than the energy threshold and is before the audio frame in the middle stage of pronunciation in the multiple audio frames as the audio frame in the beginning stage of pronunciation; label the audio frame whose audio frame energy is less than the energy threshold and is after the audio frame in the middle stage of pronunciation in the multiple audio frames as the audio frame in the end stage of pronunciation.

[0108] In practical applications, the user's volume is often lower at the beginning and end of the pronunciation phases than in the middle phases. Therefore, in this embodiment, an energy threshold (such as threshold a) can be preset. After identifying multiple audio frames associated with the original lyrics, the audio frame energy corresponding to each associated audio frame can be obtained, and then compared with the energy threshold.

[0109] After obtaining the comparison results of audio frame energy and energy threshold, audio frames with energy greater than or equal to the energy threshold and located in the middle of multiple audio frames are marked as audio frames in the middle stage of pronunciation. Then, on the one hand, audio frames with energy less than the energy threshold and located before the audio frames in the middle stage of pronunciation among the multiple audio frames associated with the original lyrics are marked as audio frames in the beginning stage of pronunciation. On the other hand, audio frames with energy less than the energy threshold and located after the audio frames in the middle stage of pronunciation among the multiple audio frames associated with the original lyrics are marked as audio frames in the end stage of pronunciation. Thus, by combining audio frame energy and energy threshold, the pronunciation stage of each audio frame associated with the original lyrics can be quickly and accurately identified.

[0110] In one embodiment, such as Figure 4 As shown, a method for obtaining lyrics timestamps is provided. This embodiment uses the application of this method to a server as an example for illustration. It can be understood that this method can also be applied to a terminal, and can also be applied to a system including a terminal and a server, and can be implemented through the interaction between the terminal and the server.

[0111] The server can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing cloud computing services. The terminal can be, but is not limited to, various personal computers, laptops, smartphones, tablets, IoT devices, and portable wearable devices. IoT devices can include smart speakers, smart TVs, smart air conditioners, smart in-vehicle devices, and projection devices. Portable wearable devices can include smartwatches, smart bracelets, and head-mounted displays. Head-mounted displays can include virtual reality (VR) devices, augmented reality (AR) devices, and smart glasses.

[0112] In this embodiment, the method includes the following steps S401 to S405:

[0113] S401, Obtain the song audio and the corresponding lyrics text, wherein the song audio includes multiple song audio frames and the lyrics text includes multiple lyrics text.

[0114] In some embodiments, the song audio obtained in this step may include existing song audio in the music library, such as Mandarin songs and foreign language songs. Of course, the song audio may also be song audio uploaded by the terminal device, such as newly released song audio with lyrics by the user.

[0115] In some embodiments, the song audio obtained in this step may include audio lacking lyric timestamps, or song audio with incorrect lyric timestamps, such as song audio with lyric timestamp offsets reported by users or detected by the server. It is understood that manually annotating, detecting, and correcting song audio lacking or with incorrect lyric timestamps would require a significant amount of manpower and time. In this embodiment, the server can automatically detect and correct offset lyric timestamps, or automatically detect and generate lyrics without timestamps, thus facilitating the efficient construction of an intelligent music library.

[0116] In practical applications, it is possible to obtain the audio of a song and the lyrics text of the same song audio. The lyrics text may include multiple lyrics corresponding to different times, and the song audio includes multiple song audio frames corresponding to different times. It is necessary to determine the correspondence between the lyrics text and the song audio frames.

[0117] S402, input the song audio into the trained feature extraction model, and the audio feature extraction module in the feature extraction model performs feature extraction on at least some of the song audio frames from multiple song audio frames to obtain the lyrics text features corresponding to each song audio frame in at least some of the song audio frames.

[0118] The feature extraction model is trained using any of the feature extraction model training methods mentioned above.

[0119] In this embodiment, a feature extraction model can be pre-trained. This feature extraction model can be used to extract audio features and text features. For two types of input information with different modalities, namely audio and text, since audio and text with different modalities can represent the same or similar semantics, the trained feature extraction model can determine the mapping relationship between the audio feature space and the text feature space. Thus, the audio features in the audio feature space can be mapped to the text feature space to obtain the text features corresponding to the audio features, and vice versa.

[0120] Furthermore, after obtaining the song audio, the song audio can be input into a trained feature extraction model, which will determine the audio features of at least some of the song audio frames and use them as the text features of the lyrics corresponding to the song audio frames.

[0121] In some embodiments, the song audio can be segmented, and the resulting multiple audio frames can be input as song audio frames into a feature extraction model. Alternatively, the song audio can be input into the feature extraction model, which will segment the audio frames to obtain multiple song audio frames, and then the audio frame features corresponding to each song audio frame can be obtained. In another embodiment, human voice detection can be performed on the segmented audio frames. Since the part of the song audio corresponding to the lyrics actually contains human voices, by performing human voice detection on the segmented audio frames, the song audio frames can be obtained based on the audio frames containing human voices, thereby reducing the data processing volume of the feature extraction model and improving the efficiency of lyrics timestamp generation.

[0122] S403, from the pre-built feature dictionary, obtain the text features of each lyric text corresponding to multiple lyric texts; the text features of the lyric texts in the feature dictionary are extracted by the text feature extraction module in the feature extraction model.

[0123] In practical applications, a feature dictionary can be pre-constructed. This dictionary can include multiple lyric text features, each corresponding to a single lyric character. These features are extracted by the text feature extraction module within the feature extraction model. When a trigger event for generating the lyric timestamp is detected, the text features corresponding to each of the multiple lyric characters can be retrieved from the feature dictionary.

[0124] In one embodiment, the feature dictionary can be constructed using the following steps:

[0125] The sample lyrics text is input into the text feature extraction module of the trained feature extraction model. The text feature extraction module obtains the lyrics text features corresponding to the lyrics text in the sample lyrics text. Based on the lyrics text features corresponding to the lyrics text in the sample lyrics text, a feature dictionary is constructed.

[0126] Specifically, after training the feature extraction model, sample lyrics text can be input into the text feature extraction module of the model. In some embodiments, the sample lyrics text can be input into the text feature extraction module on a character-by-character or sentence-by-sentence basis. Then, based on the output of the text feature extraction module, the lyrics text features corresponding to the lyrics characters in the sample lyrics text can be obtained. Afterwards, a feature dictionary can be generated based on the lyrics text features corresponding to each lyrics character in the sample lyrics text. By pre-constructing a feature dictionary based on the lyrics text features corresponding to the lyrics characters in the sample lyrics text, the repeated generation of lyrics text features for the same lyrics characters can be avoided, effectively saving computational resources and improving the efficiency of lyrics timestamp acquisition.

[0127] S404, determine the lyrics text corresponding to each song audio frame based on the similarity between the lyrics text features corresponding to each song audio frame and the lyrics text features corresponding to each lyrics text.

[0128] Specifically, on the one hand, this embodiment analyzes multiple song audio frames through the audio feature extraction module in the feature extraction model to obtain the lyrics text features corresponding to each song audio frame. On the other hand, the lyrics text features in the feature dictionary are extracted by the text feature extraction module in the feature extraction model. Since the model training of the feature extraction model has enabled the audio feature extraction module and the text feature extraction module to output the same or similar feature extraction results for audio and text representing the same semantics, and output different feature extraction results for audio and text representing different semantics, in this step, for the lyrics text features corresponding to each audio frame, the lyrics text features can be compared with the lyrics text features obtained in step S403 to determine the similarity between the lyrics text features corresponding to each song audio frame and the lyrics text features corresponding to each lyrics text. Then, the lyrics text corresponding to each song audio frame is determined based on the similarity.

[0129] For example, when text features are represented by embedding, the distance between the text features of the lyrics corresponding to the audio frame of a song and the text features of the lyrics corresponding to the lyrics can be determined. The similarity is determined based on the distance, where the similarity is inversely proportional to the distance. Thus, the lyrics with the highest similarity can be taken as the lyrics corresponding to the audio frame of the song.

[0130] S405: Obtain the lyrics timestamp of the song audio based on the lyrics text corresponding to each audio frame of the song.

[0131] The above steps allow you to obtain the lyrics corresponding to each audio frame of a song. By combining the lyrics from each audio frame, you can obtain the lyrics timestamp. For example, you can determine the lyrics timestamp based on the lyrics and the timing of each audio frame.

[0132] In this embodiment, the text feature extraction module and audio feature extraction module in the feature extraction model can extract features from lyrics text and song audio frames of different modalities, respectively. Since the text feature extraction module and audio feature extraction module have been pre-trained to output the same or similar feature expressions for text and audio representing the same semantics, this embodiment can efficiently and accurately determine the lyrics text corresponding to the song audio frame by comparing the similarity of lyrics text features. Compared with the traditional method of converting text into phonemes and then into HMM state sequences, and then performing Viterbi decoding based on state transition probabilities, this embodiment does not need to introduce an additional phoneme dictionary, but directly aligns audio and text at the word level. This not only makes the process simpler, but also has better results, and can effectively improve the alignment accuracy of the finally obtained lyrics timestamps.

[0133] Furthermore, manually creating lyric timestamps has a certain threshold and is a challenge for relevant users. The lyric timestamp acquisition method in this embodiment can help musicians simplify the process of releasing their works and lower the threshold for musicians to create. After obtaining the text features of multiple characters in the text feature space, they can be reused to quickly match the corresponding lyric text and generate lyric timestamps for song audio of different language types. This effectively covers the musicians' production process and attracts more users to create music.

[0134] To enable those skilled in the art to better understand the above steps, the following is combined with... Figure 2 The embodiments of this application are illustrated by way of example, but it should be understood that the embodiments of this application are not limited thereto.

[0135] like Figure 2As shown, first, obtain the sample lyric text and sample song audio for training the model, align them through the lyric timestamp model to obtain multiple audio frames associated with each original lyric word, and use the multiple audio frames associated with the original lyric word as the multiple audio frames associated with the corresponding sample lyric word. Taking the original lyric word "我" as an example, through the lyric timestamp model, it is possible to determine 6 audio frames associated with the original lyric word "我", and use these 6 audio frames as the 6 audio frames associated with the sample lyric word "我_起始".

[0136] Then, it is possible to perform pronunciation stage recognition and annotation on the 6 audio frames associated with the original lyric word "我". According to the pronunciation stage recognition results, label the 1st and 2nd audio frames among the 6 audio frames as the audio frames in the pronunciation start stage, label the 3rd and 4th audio frames among the 6 audio frames as the audio frames in the pronunciation middle stage, and label the 5th and 6th audio frames among the 6 audio frames as the audio frames in the pronunciation end stage.

[0137] For the 3rd and 4th audio frames labeled as the pronunciation middle stage, use them as the sample audio frames corresponding to the sample lyric word "我_中间". Subsequently, one of the sample audio frames can be combined with the sample lyric word "我_中间" to form a positive example.

[0138] For the pronunciation start stage or pronunciation end stage, taking the pronunciation start stage as an example for illustration, only the 1st and 2nd audio frames labeled as the pronunciation start stage can be used as the sample audio frames corresponding to the sample lyric word "我_起始"; of course, it can also be expanded. In some embodiments, the multiple audio frames associated with two consecutive original lyric words are not consecutive. As Figure 2 所 shown, there are still several unpartitioned audio frames between the 6 audio frames associated with the original lyric word "我" and the 6 audio frames associated with the original lyric word "爱". In addition to using the 1st and 2nd audio frames labeled as the pronunciation start stage as the sample audio frames corresponding to the sample lyric word "我_起始", also use 1 audio frame before the audio frames labeled as the pronunciation start stage as the sample audio frame. By expanding the audio frames in the pronunciation start stage and pronunciation end stage, it is possible to correctly identify the audio frames that may be misrecognized as environmental sounds but are actually related to the lyric words, avoiding omission. Figure 2 The processing of the original lyric words "爱" and "你" in is similar to the processing of the original lyric word "我", and will not be elaborated here.

[0139] Then, input the sample audio frames into the audio feature extraction module to extract audio features; on the other hand, input the sample lyric words into the text feature extraction module to extract text features, input the two-way representations into two fully connected layers respectively, calculate the contrast loss value at the output end of the fully connected layer, and optimize the loss function to train until the model converges to obtain the feature extraction model.

[0140] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0141] In one exemplary embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 5 As shown, this computer device includes a processor, memory, input / output (I / O) interfaces, and a communication interface. The processor, memory, and I / O interfaces are connected via a system bus, and the communication interface is also connected to the system bus via the I / O interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides the environment for the operating system and computer programs stored in the non-volatile storage media. The database stores audio song data and lyrics text data. The I / O interfaces are used for exchanging information between the processor and external devices. The communication interface is used for communicating with external terminals via a network connection. When executed by the processor, the computer program implements a lyrics timestamp acquisition method.

[0142] In one exemplary embodiment, a computer device is provided, which may be a terminal, and its internal structure diagram may be as follows: Figure 6As shown, the computer device includes a processor, memory, input / output interfaces, a communication interface, a display unit, and an input device. The processor, memory, and input / output interfaces are connected via a system bus, and the communication interface, display unit, and input device are also connected to the system bus via the input / output interfaces. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The input / output interfaces are used for exchanging information between the processor and external devices. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, mobile cellular networks, Near Field Communication (NFC), or other technologies. When executed by the processor, the computer program implements a method for obtaining lyrics timestamps. The display unit is used to form a visually visible image and can be a display screen, a projection device, or a virtual reality imaging device. The display screen can be an LCD screen or an e-ink screen. The input device of the computer device can be a touch layer covering the display screen, or buttons, trackballs, or touchpads set on the casing of the computer device, or external keyboards, touchpads, or mice, etc.

[0143] Those skilled in the art will understand that Figure 5 and Figure 6 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0144] In one embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above-described method embodiments.

[0145] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the steps in the above method embodiments.

[0146] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps in the above method embodiments.

[0147] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, played data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data must comply with relevant regulations.

[0148] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, artificial intelligence (AI) processors, etc., and are not limited to these.

[0149] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0150] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A method for training a feature extraction model, characterized in that, The method includes: Obtain sample song audio and multiple sample lyrics texts corresponding to the sample song audio; wherein the sample lyrics texts are obtained by annotating the pronunciation stages of the original lyrics texts in the sample lyrics texts corresponding to the sample song audios, and the sample lyrics texts are associated with pronunciation stage information; Based on the pronunciation stage of each sample lyric, determine the sample audio frame corresponding to each sample lyric from the sample song audio; Based on the multiple sample audio frames and the sample lyrics text corresponding to each sample audio frame, positive sample pairs and negative sample pairs are determined; the positive sample pair includes the sample audio frame and the sample lyrics text corresponding to the sample audio frame, and the negative sample pair includes the sample audio frame and the sample lyrics text that does not correspond to the sample audio frame. The audio feature extraction module obtains the audio features of each of the sample audio frames in the multiple sample audio frames, and the text feature extraction module determines the text features of each of the sample lyrics text. The contrast loss value is determined based on the first feature difference and the second feature difference; the first feature difference is the difference between the audio features of the sample audio frame and the text features of the sample lyrics in the positive sample pair, and the second feature difference is the difference between the audio features of the sample audio frame and the text features of the sample lyrics in the negative sample pair. The module parameters of the audio feature extraction module and the text feature extraction module are adjusted based on the contrast loss value to obtain the trained feature extraction model.

2. The method according to claim 1, characterized in that, The step of adjusting the module parameters of the audio feature extraction module and the text feature extraction module based on the contrast loss value includes: The parameters of the audio feature extraction module and the text feature extraction module are adjusted based on the contrast loss value using the backpropagation algorithm.

3. The method according to claim 1, characterized in that, The step of determining the sample audio frame corresponding to each sample lyric character from the sample song audio based on the pronunciation stage of each sample lyric character includes: Based on the timestamp of the lyrics corresponding to the sample lyrics text, determine multiple audio frames associated with each original lyric text from the sample song audio. For each sample lyric text, the multiple audio frames associated with the original lyric text corresponding to the sample lyric text are taken as the multiple audio frames associated with the sample lyric text. For each sample lyric text, the sample audio frame corresponding to the sample lyric text is determined from the sample song audio based on the pronunciation stage of the sample lyric text and the pronunciation stage of each audio frame in the multiple audio frames associated with the sample lyric text.

4. The method according to claim 3, characterized in that, The pronunciation stage can be the beginning stage, the middle stage, or the end stage; determining the sample audio frame corresponding to the sample lyrics from the sample song audio based on the pronunciation stage of the sample lyrics and the pronunciation stage of each audio frame in the multiple audio frames associated with the sample lyrics includes: If the pronunciation stage of the sample lyrics is the middle stage of pronunciation, the audio frame whose pronunciation stage is marked as the middle stage of pronunciation among the multiple audio frames associated with the sample lyrics is determined as the sample audio frame corresponding to the sample lyrics. If the pronunciation stage of the sample lyrics is the pronunciation start stage, the audio frame whose pronunciation stage is marked as the pronunciation start stage among the multiple audio frames associated with the sample lyrics is determined as the sample audio frame corresponding to the sample lyrics. If the pronunciation stage of the sample lyrics is the end of pronunciation, the audio frame whose pronunciation stage is marked as the end of pronunciation among the multiple audio frames associated with the sample lyrics is determined as the sample audio frame corresponding to the sample lyrics.

5. The method according to claim 3, characterized in that, The pronunciation stage can be the beginning stage, the middle stage, or the end stage; determining the sample audio frame corresponding to the sample lyrics from the sample song audio based on the pronunciation stage of the sample lyrics and the pronunciation stage of each audio frame in the multiple audio frames associated with the sample lyrics includes: If the pronunciation stage of the sample lyrics is the middle stage of pronunciation, the audio frame marked as the middle stage of pronunciation among the multiple audio frames associated with the sample lyrics is determined as the sample audio frame corresponding to the sample lyrics. If the pronunciation stage of the sample lyrics is the pronunciation start stage, the audio frames marked as the pronunciation start stage in the multiple audio frames associated with the sample lyrics, as well as a preset number of audio frames before the pronunciation start stage audio frames in the sample song audio, are determined as the sample audio frames corresponding to the sample lyrics. If the pronunciation stage of the sample lyrics is the end of pronunciation, the audio frames marked as the end of pronunciation stage in the multiple audio frames associated with the sample lyrics, and a preset number of audio frames after the audio frames of the end of pronunciation stage in the sample song audio, are determined as the sample audio frames corresponding to the sample lyrics.

6. The method according to claim 3, characterized in that, The pronunciation stage can be the pronunciation initiation stage, the pronunciation middle stage, or the pronunciation end stage; the method further includes: Obtain the audio frame energy of each audio frame in the multiple audio frames associated with the original lyrics text; Audio frames whose energy is greater than or equal to the energy threshold and are in the middle of multiple audio frames are labeled as audio frames in the middle stage of pronunciation. Audio frames whose energy is less than the energy threshold and which are in the middle stage of pronunciation among multiple audio frames are marked as audio frames in the beginning stage of pronunciation. Audio frames whose energy is less than the energy threshold and which are in the middle stage of pronunciation among multiple audio frames are marked as audio frames in the end stage of pronunciation.

7. A method for obtaining lyrics timestamps, characterized in that, The method includes: Obtain the song audio and the corresponding lyrics text, wherein the song audio includes multiple song audio frames and the lyrics text includes multiple lyrics text; The song audio is input into a trained feature extraction model, and the audio feature extraction module in the feature extraction model performs feature extraction on at least some of the song audio frames in the plurality of song audio frames to obtain the lyrics text features corresponding to each of the at least some of the song audio frames. The feature extraction model is trained by the feature extraction model training method of any one of claims 1 to 6. The text features of each lyric text in the plurality of lyric texts are obtained from the pre-constructed feature dictionary; the text features of the lyric texts in the feature dictionary are extracted by the text feature extraction module in the feature extraction model. Based on the similarity between the text features of the lyrics corresponding to each audio frame of the song and the text features of the lyrics corresponding to each lyric text, the lyrics text corresponding to each audio frame of the song is determined; Based on the lyrics text corresponding to each audio frame of the song, obtain the lyrics timestamp of the song audio.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the feature extraction model training method according to any one of claims 1 to 6 or the steps of the lyrics timestamp acquisition method according to claim 7.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the steps of the feature extraction model training method according to any one of claims 1 to 6 or the steps of the lyrics timestamp acquisition method according to claim 7.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the steps of the feature extraction model training method according to any one of claims 1 to 6 or the steps of the lyrics timestamp acquisition method according to claim 7.

Images