Feature extraction model training method, lyric timestamp obtaining method and related products

Abstract

The invention relates to a feature extraction model training method, a lyric timestamp acquisition method, computer equipment, a readable storage medium and a program product, relates to the technical field of audio processing, and can improve the alignment accuracy of song audio and text. The feature extraction model training method comprises the following steps: acquiring a sample song audio and a plurality of sample lyric characters corresponding to the sample song audio; according to the pronunciation stage of each sample lyric character, determining a sample audio frame corresponding to each sample lyric character from the sample song audio; determining a positive sample pair and a negative sample pair according to the plurality of sample audio frames and the sample lyric characters corresponding to each sample audio frame; performing comparative learning training on an audio feature extraction module and a text feature extraction module according to the positive sample pair and the negative sample pair to obtain a trained feature extraction model; wherein the audio feature extraction module is used for extracting audio features of audio frames; the text feature extraction module is used for extracting text features of lyric characters.

Description

Technical Field

[0001] The present application relates to the field of audio processing technology, and in particular to a feature extraction model training method, a lyrics timestamp acquisition method, a computer device, a computer-readable storage medium, and a computer program product. Background Art

[0002] With the development of computer technology, it is possible to quickly align song audio with the corresponding text content by automatically generating lyrics timestamps.

[0003] In the related technology, the timestamps of lyrics are automatically generated by combining acoustic models and statistical models. The acoustic model is responsible for learning the correspondence between states and audio frames, and the statistical model is responsible for determining the causal relationship between states. The two are combined to build an automatic alignment system for song audio and text. However, it is found in practice that the alignment accuracy of song audio and text in the above method still needs to be improved. Summary of the invention

[0004] Based on this, it is necessary to provide a feature extraction model training method, a lyrics timestamp acquisition method, a computer device, a computer-readable storage medium and a computer program product to address the above technical problems.

[0005] In a first aspect, the present application provides a method for obtaining lyrics timestamps, comprising:

[0006] Obtain a sample song audio and a plurality of sample lyrics corresponding to the sample song audio; wherein the sample lyrics are obtained after annotating the pronunciation of the original lyrics in the sample lyrics text corresponding to the sample song audio;

[0007] According to the pronunciation stage of each of the sample lyrics, determine the sample audio frame corresponding to each of the sample lyrics from the sample song audio;

[0008] According to the plurality of sample audio frames and the sample lyrics text corresponding to each of the sample audio frames, a positive sample pair and a negative sample pair are determined; the positive sample pair includes the sample audio frame and the sample lyrics text corresponding to the sample audio frame, and the negative sample pair includes the sample audio frame and the sample lyrics text not corresponding to the sample audio frame;

[0009] According to the positive sample pairs and the negative sample pairs, comparative learning training is performed on the audio feature extraction module and the text feature extraction module to obtain a trained feature extraction model; wherein the audio feature extraction module is used to extract audio features of audio frames; and the text feature extraction module is used to extract text features of lyrics.

[0010] In one embodiment, the audio feature extraction module and the text feature extraction module are subjected to comparative learning training based on the positive sample pair and the negative sample pair to obtain a trained feature extraction model, including:

[0011] The audio feature extraction module obtains the audio feature of each of the plurality of sample audio frames, and the text feature extraction module determines the text feature of each of the sample lyrics text;

[0012] Determine the contrast loss value according to the first feature difference and the second feature difference; the first feature difference is the difference between the audio feature of the sample audio frame in the positive sample pair and the text feature of the sample lyrics, and the second feature difference is the difference between the audio feature of the sample audio frame in the negative sample pair and the text feature of the sample lyrics;

[0013] The module parameters of the audio feature extraction module and the text feature extraction module are adjusted according to the contrast loss value to obtain a trained feature extraction model.

[0014] In one embodiment, the step of determining the sample audio frame corresponding to each sample lyric text from the sample song audio according to the pronunciation stage of each sample lyric text comprises:

[0015] Determining multiple audio frames associated with each of the original lyrics texts from the sample song audio according to the lyrics timestamps corresponding to the sample lyrics texts;

[0016] For each of the sample lyrics, multiple audio frames associated with the original lyrics corresponding to the sample lyrics are used as multiple audio frames associated with the sample lyrics;

[0017] For each of the sample lyrics, the sample audio frame corresponding to the sample lyrics is determined from the sample song audio according to the pronunciation stage of the sample lyrics and the pronunciation stage of each audio frame in the multiple audio frames associated with the sample lyrics.

[0018] In one embodiment, the pronunciation stage is a pronunciation start stage, a pronunciation middle stage or a pronunciation end stage; and determining the sample audio frame corresponding to the sample lyrics from the sample song audio according to the pronunciation stage of the sample lyrics and the pronunciation stage of each audio frame in the multiple audio frames associated with the sample lyrics, comprises:

[0019] If the pronunciation stage of the sample lyrics is the middle stage of pronunciation, determine the audio frame whose pronunciation stage is marked as the middle stage of pronunciation among the multiple audio frames associated with the sample lyrics as the sample audio frame corresponding to the sample lyrics;

[0020] If the pronunciation stage of the sample lyrics is the pronunciation start stage, determine the audio frame whose pronunciation stage is marked as the pronunciation start stage among the multiple audio frames associated with the sample lyrics as the sample audio frame corresponding to the sample lyrics;

[0021] If the pronunciation stage of the sample lyrics is the pronunciation end stage, the audio frame whose pronunciation stage is marked as the pronunciation end stage among the multiple audio frames associated with the sample lyrics is determined as the sample audio frame corresponding to the sample lyrics.

[0022] In one embodiment, the pronunciation stage is a pronunciation start stage, a pronunciation middle stage or a pronunciation end stage; and determining the sample audio frame corresponding to the sample lyrics from the sample song audio according to the pronunciation stage of the sample lyrics and the pronunciation stage of each audio frame in the multiple audio frames associated with the sample lyrics, comprises:

[0023] If the pronunciation stage of the sample lyrics is an intermediate pronunciation stage, determining the audio frame marked as the intermediate pronunciation stage among the multiple audio frames associated with the sample lyrics as the sample audio frame corresponding to the sample lyrics;

[0024] If the pronunciation stage of the sample lyrics is the pronunciation start stage, the audio frames marked as the pronunciation start stage among the multiple audio frames associated with the sample lyrics and the preset number of audio frames before the audio frames in the pronunciation start stage in the sample song audio are determined as the sample audio frames corresponding to the sample lyrics;

[0025] If the pronunciation stage of the sample lyrics text is the pronunciation end stage, the audio frames marked as the pronunciation end stage among the multiple audio frames associated with the sample lyrics text, and a preset number of audio frames after the audio frames in the pronunciation end stage in the sample song audio are determined as sample audio frames corresponding to the sample lyrics text.

[0026] In one embodiment, the pronunciation stage is a pronunciation start stage, a pronunciation middle stage or a pronunciation end stage; the method further comprises:

[0027] Obtaining the audio frame energy of each audio frame in a plurality of audio frames associated with the original lyrics text;

[0028] The audio frame energy is greater than or equal to the energy threshold and the audio frame in the middle of the multiple audio frames is marked as an audio frame in the middle stage of pronunciation;

[0029] Marking the audio frame whose energy is less than the energy threshold and before the audio frame in the middle stage of pronunciation among the multiple audio frames as an audio frame in the initial stage of pronunciation;

[0030] The audio frame energy of the audio frame is less than the energy threshold and the audio frame after the audio frame in the middle stage of pronunciation among the multiple audio frames is marked as an audio frame in the end stage of pronunciation.

[0031] In a second aspect, the present application also provides a method for obtaining lyrics timestamps, the method comprising:

[0032] Obtain song audio and lyrics text corresponding to the song audio, wherein the song audio includes a plurality of song audio frames, and the lyrics text includes a plurality of lyrics characters;

[0033] The song audio is input into a trained feature extraction model, and an audio feature extraction module in the feature extraction model performs feature extraction on at least part of the multiple song audio frames to obtain lyrics text features corresponding to each of the at least part of the song audio frames, wherein the feature extraction model is trained by the feature extraction model training method as described in any one of the above items;

[0034] Obtaining lyrics text features corresponding to each of the multiple lyrics texts from a pre-constructed feature dictionary; the lyrics text features in the feature dictionary are extracted by a text feature extraction module in the feature extraction model;

[0035] Determine the lyrics text corresponding to each of the song audio frames according to the similarity between the lyrics text features corresponding to each of the song audio frames and the lyrics text features corresponding to each of the lyrics texts;

[0036] The lyrics timestamp of the song audio is obtained according to the lyrics text corresponding to each of the song audio frames.

[0037] In a third aspect, the present application also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the steps of the feature extraction model training method or the steps of the lyrics timestamp acquisition method described in any one of the above items are implemented.

[0038] In a fourth aspect, the present application also provides a computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the steps of the feature extraction model training method or the steps of the lyrics timestamp acquisition method described in any one of the above items are implemented.

[0039] In a fifth aspect, the present application also provides a computer program product, including a computer program, which, when executed by a processor, implements the steps of any of the above-mentioned feature extraction model training methods or the steps of the above-mentioned lyrics timestamp acquisition methods.

[0040] The above-mentioned feature extraction model training method determines the sample audio frame corresponding to each sample lyric text from the sample song audio according to the pronunciation stage of each sample lyric text; determines the positive sample pair and the negative sample pair according to multiple sample audio frames and the sample lyric text corresponding to each sample audio frame, wherein the positive sample pair includes the sample audio frame and the sample lyric text corresponding to the sample audio frame, and the negative sample pair includes the sample audio frame and the sample lyric text that does not correspond to the sample audio frame; and then, based on the positive sample pair and the negative sample pair, performs comparative learning and training on the audio feature extraction module and the text feature extraction module to obtain a feature extraction model. This training method enables the trained feature extraction model to learn the mapping relationship between the audio feature space and the text feature space. This mapping relationship is applied in the lyrics text alignment scenario to more accurately align the song audio and song text. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the drawings required for use in the embodiments of the present application or related technical descriptions will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0042] Figure 1 A schematic diagram of a flow chart of a feature extraction model training method in an embodiment;

[0043] Figure 2 A schematic diagram of a flow chart of another method for training a feature extraction model in one embodiment;

[0044] Figure 3 A schematic flow chart of a step of determining a sample audio frame corresponding to a sample lyric text in one embodiment;

[0045] Figure 4 A schematic diagram of a process for obtaining lyrics timestamps in one embodiment;

[0046] Figure 5 is an internal structure diagram of a computer device in one embodiment;

[0047] Figure 6 The figure is a diagram of the internal structure of another computer device in one embodiment. DETAILED DESCRIPTION

[0048] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0049] In order to enable those skilled in the art to better understand the present application, the relevant technologies are introduced below.

[0050] With the development of computer technology, it is possible to quickly align song audio with the corresponding text content by automatically generating lyrics timestamps. Automatically generating lyrics timestamps means obtaining the start and end time of each pronunciation corresponding to the word in the song audio through an automatic alignment algorithm for the input song audio and the corresponding text content. By automatically aligning lyrics, the cost of manual annotation of lyrics timestamps will be greatly reduced, and the production process of musicians will also be simplified.

[0051] In the related technology, the lyrics timestamp is mainly automatically generated by combining acoustic models and statistical models. In one example, the lyrics timestamp model adopts a modeling scheme based on DNN (Deep Neural Network)-HMM (Hidden Markov Model). In this scheme, the DNN model is responsible for learning the correspondence between states and frames, and the HMM model is responsible for the causal correlation between states. The two are combined to build a non-end-to-end automatic alignment system.

[0052] Specifically, in the alignment system, on the one hand, the spectral features are first extracted from the song audio and input into the DNN model, and the predicted probability of each category is output in the classification layer (such as the Softmax layer) of the DNN model, and the number of output categories is consistent with the number of states in the state space. On the other hand, the system converts the lyrics text into a phoneme sequence. For example, a text-to-phoneme conversion dictionary can be obtained in advance, and then the lyrics text is converted into a phoneme sequence containing multiple phonemes through the conversion dictionary. For Chinese, pinyin can be used, and for English, international phonetic symbols can be used; and then after obtaining the phoneme sequence, the phoneme sequence can be converted into a state sequence of HMM. Generally speaking, a phoneme corresponds to several states (such as three states), which can be self-transferred or transferred to other states. The specific state transition probability is described by the hidden Markov model. In this example, the state can be understood as the minimum granularity of a word, and one frame of audio features corresponds to one state.

[0053] Then, for the probability that each frame outputted previously belongs to each state in the state space, HMM outputs the state transition probability, obtains the path with the highest global probability through Viterbi decoding, aligns the frames and states one by one to obtain the decoding result (that is, the state corresponding to each frame), and then converts the state corresponding to each frame into phonemes and then converts the phonemes into words to obtain the lyrics timestamp.

[0054] However, it is found in practice that the alignment accuracy of song audio and text in the above method is low, and there is still a lot of room for improvement. Based on this, the present application provides a feature extraction model training method, a lyrics timestamp acquisition method, a computer device, a readable storage medium and a program product to at least solve the above technical problems.

[0055] In one embodiment, Figure 1 As shown, a feature extraction model training method is provided. This embodiment uses the method applied to a server as an example. It can be understood that the method can also be applied to a terminal, and can also be applied to a system including a terminal and a server, and is implemented through the interaction between the terminal and the server. In this embodiment, the feature extraction model training method includes the following steps S101 to S104:

[0056] S101, obtaining a sample song audio and a plurality of sample lyrics texts corresponding to the sample song audio; wherein the sample lyrics texts are obtained after the pronunciation stage of the original lyrics texts in the sample lyrics texts corresponding to the sample song audio.

[0057] In a specific implementation, song audio for training a feature extraction model may be obtained. To facilitate distinction, the song audio for training the model is referred to as sample song audio. The sample song audio has corresponding sample lyrics text, and the sample lyrics text includes one or more lyrics texts, which are the original lyrics texts in the sample lyrics. To distinguish them from other lyrics texts, the lyrics texts are referred to as original lyrics texts.

[0058] During the singing process, the acoustic features of the same original lyrics text at different pronunciation stages may be different. For example, due to the singer's breath changes or the use of singing transition techniques, the acoustic features of the same original lyrics text at the beginning, middle and end stages of the singing process may be different. In this regard, in this embodiment, the original lyrics text can be pre-labeled with the pronunciation stage to obtain the corresponding sample lyrics text. Different from the original lyrics text without the pronunciation stage, the sample lyrics text is the lyrics text labeled with the pronunciation stage. For example, for the original lyrics text "I", after labeling the pronunciation stage of the text, three sample lyrics texts can be obtained, namely "I_start", "I_middle" and "I_end".

[0059] S102, determining a sample audio frame corresponding to each sample lyric text from the sample song audio according to the pronunciation stage of each sample lyric text.

[0060] After obtaining multiple sample lyrics, the sample song audio can be analyzed to determine the audio frame related to the sample lyrics and matching the pronunciation stage from the sample song audio. The audio frame is also called the sample audio frame corresponding to the sample lyrics. For example, an audio frame containing an acoustic signal corresponding to the original lyrics and whose pronunciation stage matches the pronunciation stage marked by the sample lyrics can be used as the sample audio frame corresponding to the sample lyrics.

[0061] In some examples, for each sample lyric text corresponding to the original lyric text (that is, the sample lyric text obtained by marking the pronunciation stage of the original lyric text), the audio frame of the original lyric text within the corresponding time range can be determined based on the start time and end time of the lyric text indicated by the timestamp in the sample lyric text, and then the sample audio frame corresponding to the sample lyric text can be determined from the audio frames within the time range or the audio frames adjacent to the time range.

[0062] S103, determining positive sample pairs and negative sample pairs according to multiple sample audio frames and sample lyrics text corresponding to each sample audio frame; the positive sample pair includes the sample audio frame and the sample lyrics text corresponding to the sample audio frame, and the negative sample pair includes the sample audio frame and the sample lyrics text that does not correspond to the sample audio frame.

[0063] After determining the sample audio frame corresponding to each sample lyric text, the sample lyric text corresponding to each sample audio frame can be obtained based on the correspondence between the sample lyric text and the sample audio frame, and then positive sample pairs and negative sample pairs can be constructed based on the sample lyric text corresponding to each sample audio frame. In other words, in the present embodiment, pairs are formed in units of single sample audio frames, and a sample pair may include a sample audio frame and a sample lyric text, wherein a positive sample pair is also called a positive example, which refers to a training sample constructed from a sample audio frame and the sample lyric text corresponding to the sample audio frame, and a negative sample pair is also called a negative example, which refers to a training sample constructed from a sample audio and sample lyric text that does not correspond to the sample audio frame, wherein the sample lyric text that does not correspond to the sample audio frame is lyrics text other than the sample lyric text corresponding to the sample audio frame.

[0064] by Figure 2For example, a sample lyric text "我_起始" (I_start) and a sample audio frame corresponding to "我_起始" can form a positive sample pair. Any sample audio frame corresponding to the sample lyric text "我_起始" and "爱_起始" (love_start), or any sample audio frame corresponding to the sample lyric text "我_起始" and "我_结束" (I_end) can form a negative sample pair. That is, for each sample audio frame, the sample audio frame and its corresponding sample lyric text form a positive example, and the sample audio frame and any other non-corresponding sample lyric text form one or more negative examples.

[0065] S104. According to the positive sample pairs and negative sample pairs, perform contrastive learning training on the audio feature extraction module and the text feature extraction module to obtain a trained feature extraction model. The audio feature extraction module is used to extract the audio features of audio frames, and the text feature extraction module is used to extract the text features of lyric texts.

[0066] After constructing multiple positive sample pairs and multiple negative sample pairs, the multiple positive sample pairs and multiple negative sample pairs can be used to perform self-supervised contrastive learning training on the audio feature extraction module and the text feature extraction module, so that for audio and text with the same or similar semantics, the feature extraction results output by the two modules can be as close as possible, while for audio and text with different or irrelevant semantics, the feature extraction results output by the two modules can be as different as possible. Thus, when the training end condition is met, for audio and text representing the same semantics, the features output by the two modules are the same or relatively similar. Therefore, the features in different feature spaces can be transformed and mapped to obtain the mapping relationship between the text feature space and the audio feature space.

[0067] In the trained feature extraction model, the mapping relationship between the audio feature space corresponding to the audio feature extraction module and the text feature space corresponding to the text feature extraction module has been constructed. Subsequently, the trained feature extraction model can be used to extract the audio features of audio frames and the text features of texts. Since the mapping relationship between the audio features and the text features has been determined, the corresponding text features can be quickly determined according to the audio features, and thus the audio frames and the corresponding lyric texts can be directly aligned at the character dimension.

[0068] In some optional embodiments, the audio feature extraction module and the text feature extraction module can be implemented by a large model. For example, for audio, the audio feature extraction module can be constructed by a large model MERT (Music Understanding Model with Large-Scale Self-supervised Training, an acoustic music understanding model based on large-scale self-supervised training) or other large models; for text, the text feature extraction module can be constructed by a large model BERT (Bidirectional Encoder Representations from Transformers) or other large models. In the contrastive learning training process, the training objectives may include: for audio and text that are related, minimizing the difference in features output by the audio feature extraction module and the text feature extraction module, and for irrelevant audio and text, maximizing the difference in features output by the audio feature extraction module and the text feature extraction module.

[0069] In this embodiment, according to the pronunciation stage of each sample lyric text, the sample audio frame corresponding to each sample lyric text is determined from the sample song audio; according to multiple sample audio frames and the sample lyric text corresponding to each sample audio frame, positive sample pairs and negative sample pairs are determined, wherein the positive sample pair includes the sample audio frame and the sample lyric text corresponding to the sample audio frame, and the negative sample pair includes the sample audio frame and the sample lyric text that does not correspond to the sample audio frame; then, according to the positive sample pairs and the negative sample pairs, the audio feature extraction module and the text feature extraction module are comparatively learned and trained to obtain a feature extraction model. This training method enables the trained feature extraction model to learn the mapping relationship between the audio feature space and the text feature space. This mapping relationship is applied in the lyrics-text alignment scenario to more accurately align the song audio and song text.

[0070] In one embodiment, in step S104, according to the positive sample pair and the negative sample pair, performing comparative learning training on the audio feature extraction module and the text feature extraction module to obtain a trained feature extraction model may include the following steps:

[0071] S1031, the audio feature extraction module obtains the audio feature of each sample audio frame in the plurality of sample audio frames, and the text feature extraction module determines the text feature of each sample lyric text.

[0072] Specifically, each sample audio frame and each sample lyric text may be input into an audio feature extraction module and a text feature extraction module respectively to obtain the audio feature of each sample audio frame and the text feature of each sample lyric text.

[0073] S1032, determining a contrast loss value based on a first feature difference and a second feature difference; the first feature difference is the difference between the audio features of the sample audio frame in the positive sample pair and the text features of the sample lyrics, and the second feature difference is the difference between the audio features of the sample audio frame in the negative sample pair and the text features of the sample lyrics.

[0074] After obtaining the audio features corresponding to each of the multiple sample audio frames and the text features corresponding to each of the multiple sample lyrics, the contrast loss value in the contrast learning training process is determined in units of positive sample pairs or negative sample pairs. Specifically, on the one hand, for each positive sample pair, the difference between the audio features of the sample audio frame in the positive sample pair and the text features of the sample lyrics can be determined. For the convenience of distinction, the feature difference is called the first feature difference; on the other hand, for each negative sample pair, the difference between the audio features of the sample audio frame in the negative sample pair and the features of the non-corresponding sample lyrics can be determined. For the convenience of distinction, the feature difference is called the second feature difference. Since the training purpose of contrast learning is to make the features corresponding to the audio and text with the same or similar semantics as close as possible, and the feature extraction results of the audio and text with different or irrelevant semantics as different as possible, the first feature difference and the second feature difference can be used to reflect the training conditions of the audio feature extraction module and the text feature extraction module, and calculate the corresponding contrast loss value.

[0075] S1033, adjusting the module parameters of the audio feature extraction module and the text feature extraction module according to the contrast loss value to obtain a trained feature extraction model.

[0076] In some embodiments, after obtaining the contrast loss value, the back propagation algorithm can be combined to adjust the module parameters of the audio feature extraction module and the text feature extraction module according to the contrast loss value, and then return to execute step S101, repeat the iteration multiple times until the training end condition is met to obtain a trained feature extraction model.

[0077] In this embodiment, the contrast loss value is determined according to the first feature difference and the second feature difference, and then the corresponding module parameters of the audio feature extraction module and the text feature extraction module are adjusted. The audio feature extraction module and the text feature extraction module can efficiently learn the feature representation methods related to audio and text and the feature representation methods unrelated to audio and text in a self-supervised manner, without the need to set additional training labels, and can quickly construct a mapping relationship between the audio feature space and the text feature space.

[0078] In one embodiment, in step S1032, determining the contrast loss value according to the first feature difference and the second feature difference may include the following steps: determining the first loss value according to the first feature difference; determining the second loss value according to the second feature difference; and determining the contrast loss value according to the first loss value and the second loss value. The first loss value is positively correlated with the first feature difference; and the second loss value is negatively correlated with the second feature difference.

[0079] In practical applications, the first loss value is a loss value determined according to the first feature difference, and the first loss value is positively correlated with the first feature difference, that is, the smaller the difference between the audio features of the sample audio frame and the text features of the sample lyrics in the positive sample pair, the smaller the first loss value, so that for the associated audio and text, the audio feature extraction module and the text feature extraction module can be guided to output more similar feature representations. On the other hand, the second loss value is a loss value determined according to the second feature difference, and the second loss value is negatively correlated with the second feature difference, that is, the larger the difference between the audio features of the sample audio frame and the text features of the non-corresponding sample lyrics in the negative sample pair, the smaller the second loss value, so that for the non-associated audio and text, the audio feature extraction module and the text feature extraction module can be guided to output more different feature representations, thereby increasing the discrimination of the output features for text or audio with different semantics.

[0080] The contrast loss value Lq can be determined by combining the first loss value and the second loss value. In an exemplary embodiment, the contrast loss value Lq can be determined by the following formula:

[0081]

[0082] Wherein, the numerator is the dot product of q with k+, which represents the distance between vector q and vector k+. In this example, vector q and vector k+ are the audio features and text features in the positive sample pair, and the denominator is the sum of the dot products of all positive examples and all negative examples. The parameter τ can be used to control the range of the loss function. In one example, q can be understood as the output of the audio feature extraction module in the fully connected layer, which can be represented by embedding form, and k+ is the output of the corresponding sample lyrics text in the fully connected layer of the text feature extraction module, which can be represented by embedding form.

[0083] In one embodiment, Figure 3 As shown, in step S102, according to the pronunciation stage of each sample lyric text, determining the sample audio frame corresponding to each sample lyric text from the sample song audio may include the following steps:

[0084] S301, determining multiple audio frames associated with each original lyric text from the sample song audio according to the lyrics timestamp corresponding to the sample lyric text.

[0085] In practical applications, the lyrics timestamp corresponding to the sample lyrics text can be obtained in advance. In some embodiments, the original lyrics text can be determined by the existing lyrics timestamp model, such as Figure 2 As shown, the sample song audio and the sample lyrics text corresponding to the sample song audio can be input into the existing DNN-HMM model to obtain the lyrics timestamp output by the DNN-HMM model. The lyrics timestamp includes the word timestamp of each original lyrics text in the sample lyrics text. The word timestamp can record the start time and end time of the audio signal with the original lyrics text in the sample song audio.

[0086] Then, multiple audio frames associated with each original lyric text can be determined from the sample song audio according to the lyric timestamp, such as Figure 2 As shown, the six audio frames associated with the three original lyrics "I", "love" and "you" can be determined based on the lyrics timestamp.

[0087] S302: For each sample lyric text, multiple audio frames associated with the original lyric text corresponding to the sample lyric text are used as multiple audio frames associated with the sample lyric text.

[0088] For each original lyric text in the sample lyric text, multiple sample lyric texts corresponding to the original lyric text can be obtained by marking different pronunciation stages for each original lyric text. For each sample lyric text obtained after marking, multiple audio frames associated with the original lyric text corresponding to the sample lyric text are used as multiple audio frames associated with the sample lyric text, which helps to quickly locate the sample audio frame corresponding to the sample lyric text.

[0089] S303, for each sample lyric text, according to the pronunciation stage of the sample lyric text and the pronunciation stage of each audio frame in multiple audio frames associated with the sample lyric text, determine the sample audio frame corresponding to the sample lyric text from the sample song audio.

[0090] In practical applications, multiple audio frames associated with sample lyrics can be used to identify the pronunciation phase. Figure 2 Among the three sample lyrics texts "I_start", "I_middle" and "I_end", the multiple audio frames associated with each sample lyrics text are the 6 audio frames associated with the original lyrics text "I". The pronunciation stage of these 6 audio frames can be identified to obtain the pronunciation stage of each audio frame in the multiple audio frames associated with the sample lyrics text.

[0091] In some exemplary embodiments, the pronunciation stage can be identified based on the energy of the audio frame, or it can be divided into multiple pronunciation stages according to the corresponding time sequence of the audio frame. For example, if the associated multiple audio frames are divided equally according to the number of pronunciation stages and the order of the pronunciation stages, the 6 audio frames associated with the sample lyrics text can be divided equally into 3 pronunciation stages, and each pronunciation stage includes 2 audio frames.

[0092] Then, the sample audio frame corresponding to the sample lyrics is determined by combining the pronunciation stage marked by the sample lyrics and the pronunciation stage of each audio frame in the multiple audio frames associated with the sample lyrics. Among them, the audio frames in the multiple audio frames associated with the sample lyrics that are in the same pronunciation stage as the sample lyrics can be used as the sample audio frames corresponding to the sample lyrics; or, the audio frames in the same pronunciation stage as the sample lyrics can be expanded forward and backward, and the sample audio frames corresponding to the sample lyrics can be obtained according to the expansion results.

[0093] In this embodiment, the sample audio frames corresponding to the sample lyrics text can be quickly and accurately matched according to the pronunciation stage of the sample lyrics text and the pronunciation stage of each audio frame in the multiple audio frames associated with the sample lyrics text. Model training is performed based on this correspondence, which helps the model to accurately extract audio features and text features in different pronunciation stages, thereby obtaining a reliable mapping relationship between audio features and text features in different pronunciation stages.

[0094] In one embodiment, the pronunciation stage is a pronunciation start stage, a pronunciation middle stage or a pronunciation end stage; in step S303, according to the pronunciation stage of the sample lyrics and the pronunciation stage of each audio frame in a plurality of audio frames associated with the sample lyrics, determining the sample audio frame corresponding to the sample lyrics from the sample song audio may include the following steps:

[0095] If the pronunciation stage of the sample lyrics text is the middle stage of pronunciation, the audio frames whose pronunciation stages are marked as the middle stage of pronunciation among the multiple audio frames associated with the sample lyrics text are determined as the sample audio frames corresponding to the sample lyrics text; if the pronunciation stage of the sample lyrics text is the starting stage of pronunciation, the audio frames whose pronunciation stages are marked as the starting stage of pronunciation among the multiple audio frames associated with the sample lyrics text are determined as the sample audio frames corresponding to the sample lyrics text; if the pronunciation stage of the sample lyrics text is the ending stage of pronunciation, the audio frames whose pronunciation stages are marked as the ending stage of pronunciation among the multiple audio frames associated with the sample lyrics text are determined as the sample audio frames corresponding to the sample lyrics text.

[0096] Specifically, after determining the pronunciation stage marked for each audio frame among the multiple audio frames associated with the sample lyric text, the audio frames among the multiple audio frames associated with the sample lyric text that have the same pronunciation stage as the sample lyric text can be determined as the sample audio frames corresponding to the sample lyric text. Based on this, if the pronunciation stage of the sample lyric text is the middle pronunciation stage, the audio frames among the multiple audio frames associated with the sample lyric text that are marked with the middle pronunciation stage are determined as the sample audio frames corresponding to the sample lyric text; if the pronunciation stage of the sample lyric text is the starting pronunciation stage, the audio frames among the multiple audio frames associated with the sample lyric text that are marked with the starting pronunciation stage are determined as the sample audio frames corresponding to the sample lyric text; if the pronunciation stage of the sample lyric text is the ending pronunciation stage, the audio frames among the multiple audio frames associated with the sample lyric text that are marked with the ending pronunciation stage are determined as the sample audio frames corresponding to the sample lyric text.

[0097] Continuing with Figure 2 as an example for illustration, according to the pronunciation stage recognition results of the 6 audio frames associated with the sample lyric text "I_start", the audio frames in the starting pronunciation stage are the first two audio frames (such as Figure 2 the two audio frames boxed by the dashed line), and these two audio frames can be used as the sample audio frames corresponding to the sample lyric text "I_start".

[0098] In this embodiment, the sample audio frames corresponding to the sample lyric text are directly determined from among the multiple audio frames associated with the sample lyric text, which can save the recognition time of the sample audio frames corresponding to the sample lyric text.

[0099] In another embodiment, the pronunciation stage is the starting pronunciation stage, the middle pronunciation stage, or the ending pronunciation stage; in step S203, according to the pronunciation stage of the sample lyric text and the pronunciation stage of each audio frame among the multiple audio frames associated with the sample lyric text, determining the sample audio frames corresponding to the sample lyric text from the sample song audio may include the following steps:

[0100] If the pronunciation stage of the sample lyric text is the middle pronunciation stage, the audio frames among the multiple audio frames associated with the sample lyric text that are marked with the middle pronunciation stage are determined as the sample audio frames corresponding to the sample lyric text; if the pronunciation stage of the sample lyric text is the starting pronunciation stage, the audio frames among the multiple audio frames associated with the sample lyric text that are marked with the starting pronunciation stage, and a preset number of audio frames before the audio frames in the starting pronunciation stage in the sample song audio are determined as the sample audio frames corresponding to the sample lyric text; if the pronunciation stage of the sample lyric text is the ending pronunciation stage, the audio frames among the multiple audio frames associated with the sample lyric text that are marked with the ending pronunciation stage, and a preset number of audio frames after the audio frames in the ending pronunciation stage in the sample song audio are determined as the sample audio frames corresponding to the sample lyric text.

[0101] Specifically, for the sample lyric text in the middle stage of pronunciation, the audio frames marked as the middle stage of pronunciation among the multiple audio frames associated with the sample lyric text can be directly determined as the sample audio frames corresponding to the sample lyric text.

[0102] For the audio frames marked with the start stage of pronunciation and the pronunciation stage, since the range corresponding to the audio frames is actually determined by the lyric timestamps, due to various factors, the lyric timestamps may be offset, resulting in the start time of the lyric timestamps may not truly cover the actual start time of the original lyric text, thus causing the content in the audio frames corresponding to the actual start time to be misidentified as ambient sound. Similarly, the end time of the lyric timestamps may not truly cover the actual end time of the original lyric text. For example, the end time indicated by the lyric timestamps is earlier than the actual end time, thus causing the content in the audio frames corresponding to the actual end time to be misidentified as ambient sound.

[0103] In this regard, in this embodiment, if the pronunciation stage of the sample lyric text is the start stage of pronunciation, in addition to determining the audio frames marked as the start stage of pronunciation among the multiple audio frames associated with the sample lyric text as the sample audio frames corresponding to the sample lyric text, a preset number of audio frames before the audio frames in the start stage of pronunciation in the sample song audio can also be determined as the sample audio frames corresponding to the sample lyric text. If the pronunciation stage of the sample lyric text is the end stage of pronunciation, in addition to determining the audio frames marked as the end stage of pronunciation among the multiple audio frames associated with the sample lyric text as the sample audio frames corresponding to the sample lyric text, a preset number of audio frames after the audio frames in the end stage of pronunciation in the sample song audio can also be determined as the sample audio frames corresponding to the sample lyric text.

[0104] For example, as Figure 2 shown, for the 2 audio frames in the start stage of pronunciation associated with the sample lyric text "I_start", 1 audio frame can be expanded forward to obtain 3 sample audio frames corresponding to the sample lyric text "I_start", that is, Figure 2 the 3 audio frames boxed by the solid line frame in

[0105] In this embodiment, by expanding the audio frames in the start stage of pronunciation and the end stage of pronunciation, the audio frames that may be misidentified as ambient sound but are actually related to the lyric text can be correctly recognized, avoiding omission, thereby improving the accuracy of the model in recognizing the audio frame features and text features in different pronunciation states.

[0106] In one embodiment, the pronunciation stage is the start stage of pronunciation, the middle stage of pronunciation or the end stage of pronunciation; the method may further include the following steps:

[0107] The audio frame energy of each audio frame in the multiple audio frames associated with the original lyrics text is obtained; the audio frame whose energy is greater than or equal to the energy threshold and is in the middle of the multiple audio frames is marked as an audio frame in the middle stage of pronunciation; the audio frame whose energy is less than the energy threshold and is before the audio frame in the middle stage of pronunciation among the multiple audio frames is marked as an audio frame in the starting stage of pronunciation; the audio frame whose energy is less than the energy threshold and is after the audio frame in the middle stage of pronunciation among the multiple audio frames is marked as an audio frame in the ending stage of pronunciation.

[0108] In actual applications, at the beginning and end of pronunciation, the volume of the user is often lower than that in the middle of pronunciation. Based on this, an energy threshold (such as threshold a) can be pre-set in this embodiment. After determining multiple audio frames associated with the original lyrics text, the audio frame energy corresponding to each audio frame associated with the original lyrics text can be obtained, and then the audio frame energy can be compared with the energy threshold.

[0109] After obtaining the comparison result of the audio frame energy and the energy threshold, the audio frame whose energy is greater than or equal to the energy threshold and is in the middle of multiple audio frames can be marked as an audio frame in the middle stage of pronunciation. Then, on the one hand, the audio frame whose energy is less than the energy threshold and is before the audio frame in the middle stage of pronunciation among the multiple audio frames associated with the original lyrics text is marked as an audio frame in the starting stage of pronunciation. On the other hand, the audio frame whose energy is less than the energy threshold and is after the audio frame in the middle stage of pronunciation among the multiple audio frames associated with the original lyrics text is marked as an audio frame in the ending stage of pronunciation. In this way, the audio frame energy and the energy threshold can be combined to quickly and accurately identify the pronunciation stage of each audio frame associated with the original lyrics text.

[0110] In one embodiment, Figure 4 As shown, a method for obtaining lyrics timestamp is provided. This embodiment takes the method applied to a server as an example. It can be understood that the method can also be applied to a terminal, and can also be applied to a system including a terminal and a server, and is implemented through the interaction between the terminal and the server.

[0111] The server may be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides cloud computing services; the terminal may be, but is not limited to, various personal computers, laptops, smart phones, tablet computers, IoT devices, and portable wearable devices. IoT devices may be smart speakers, smart TVs, smart air conditioners, smart car devices, and projection devices. Portable wearable devices may be smart watches, smart bracelets, head-mounted devices, etc. Head-mounted devices may be virtual reality (VR) devices, augmented reality (AR) devices, smart glasses, etc.

[0112] In this embodiment, the method includes the following steps S401 to S405:

[0113] S401, obtaining song audio and lyrics text corresponding to the song audio, wherein the song audio includes multiple song audio frames, and the lyrics text includes multiple lyrics characters.

[0114] In some embodiments, the song audio obtained in this step may include song audio already in the music library, such as Mandarin songs and foreign songs. Of course, the song audio may also be song audio uploaded by the terminal device, such as song audio newly released by the user with lyrics.

[0115] In some embodiments, the song audio obtained in this step may include audio that lacks lyrics timestamps, and may also include song audio that has incorrect lyrics timestamps, such as song audio that has lyrics timestamp offsets reported by users or detected by servers. It is understandable that for song audio that lacks lyrics timestamps or has incorrect lyrics timestamps, if manual annotation, detection, and correction are all used, it will take a lot of manpower and time. In this embodiment, the server can automatically detect and correct lyrics timestamps with offsets, or automatically detect and generate lyrics without timestamps, to help efficiently build an intelligent music library.

[0116] In actual applications, song audio and lyrics text of the same song audio can be obtained. The lyrics text may include multiple lyrics texts corresponding to different times, and the song audio includes multiple song audio frames corresponding to different times. It is necessary to determine the correspondence between the lyrics texts and the song audio frames.

[0117] S402, inputting the song audio into the trained feature extraction model, and having the audio feature extraction module in the feature extraction model perform feature extraction on at least part of the multiple song audio frames, to obtain lyrics text features corresponding to each song audio frame in at least part of the song audio frames.

[0118] The feature extraction model is trained by any of the feature extraction model training methods mentioned above.

[0119] In this embodiment, a feature extraction model can be pre-trained, and the feature extraction model can be used to extract audio features and text features. For two types of input information with different modalities, namely audio and text, since audio and text with different modalities can represent the same or similar semantics, the trained feature extraction model can determine the mapping relationship between the audio feature space and the text feature space, so that the audio features in the audio feature space can be mapped to the text feature space to obtain the text features corresponding to the audio features, and the text features in the text feature space can be mapped to the audio feature space to obtain the audio features corresponding to the text features.

[0120] Furthermore, after obtaining the song audio, the song audio can be input into a trained feature extraction model, and the feature extraction model determines the audio features of at least some of the multiple song audio frames, and uses them as the lyrics text features corresponding to the song audio frames.

[0121] In some embodiments, the song audio may be segmented, and the multiple audio frames obtained after segmentation may be input into the feature extraction model as song audio frames. Alternatively, the song audio may be input into the feature extraction model, and the model may segment the audio frames, obtain multiple song audio frames based on the segmentation results, and then obtain the audio frame features corresponding to each song audio frame. In another embodiment, human voice detection may be performed on the segmented audio frames. Since the portion of the song audio corresponding to the lyrics actually contains the human voice, by performing human voice detection on the multiple segmented audio frames, the song audio frames may be obtained based on the audio frames containing the human voice, thereby reducing the amount of data processing of the feature extraction model and improving the efficiency of generating lyrics timestamps.

[0122] S403, obtaining lyrics text features corresponding to each of the multiple lyrics texts from a pre-constructed feature dictionary; the lyrics text features in the feature dictionary are extracted by a text feature extraction module in the feature extraction model.

[0123] In practical applications, a feature dictionary can be pre-built, and the feature dictionary can include multiple lyrics text features, each of which corresponds to a lyric text. The lyrics text features in the feature dictionary are extracted by the text feature extraction module in the feature extraction model. When a trigger event for generating a lyrics timestamp is detected, the text features corresponding to each of the multiple lyrics texts in the lyrics text can be obtained from the feature dictionary.

[0124] In one embodiment, the feature dictionary can be constructed by the following steps:

[0125] The sample lyrics text is input into the text feature extraction module in the trained feature extraction model, and the text feature extraction module obtains the lyrics text features corresponding to the lyrics text in the sample lyrics text; based on the lyrics text features corresponding to the lyrics text in the sample lyrics text, a feature dictionary is constructed.

[0126] Specifically, after the feature extraction model is trained, the sample lyrics text can be input into the text feature extraction module in the feature extraction model. In some embodiments, the sample lyrics text can be input into the text feature extraction module in units of words, or in units of sentences. Then, the lyrics text features corresponding to the lyrics text in the sample lyrics text can be obtained according to the output result of the text feature extraction module. Afterwards, a feature dictionary can be generated according to the lyrics text features corresponding to each lyrics text in the sample lyrics text. By pre-building a feature dictionary based on the lyrics text features corresponding to the lyrics text in the sample lyrics text, it is possible to avoid repeatedly generating lyrics text features for the same lyrics text, effectively save computing resources, and improve the efficiency of obtaining lyrics timestamps.

[0127] S404, determining the lyrics text corresponding to each song audio frame according to the similarity between the lyrics text features corresponding to each song audio frame and the lyrics text features corresponding to each lyrics text.

[0128] Specifically, on the one hand, this embodiment analyzes multiple song audio frames through the audio feature extraction module in the feature extraction model to obtain the lyrics text features corresponding to each song audio frame. On the other hand, the lyrics text features in the feature dictionary are extracted by the text feature extraction module in the feature extraction model. Due to the previous model training of the feature extraction model, the audio feature extraction module and the text feature extraction module have output the same or similar feature extraction results for audio and text representing the same semantics, and output different feature extraction results for audio and text representing different semantics. In this step, for the lyrics text features corresponding to each audio frame, the lyrics text features can be compared with the lyrics text features obtained in step S403 to determine the similarity between the lyrics text features corresponding to each song audio frame and the lyrics text features corresponding to each lyrics text, and then the lyrics text corresponding to each song audio frame is determined based on the similarity.

[0129] For example, when text features are represented by embeddings, the distance between the text features of the lyrics corresponding to the song audio frame and the text features of the lyrics corresponding to the lyrics can be determined, and the similarity can be determined based on the distance, where the similarity is inversely proportional to the distance. The lyrics with the highest similarity can then be used as the lyrics corresponding to the song audio frame.

[0130] S405, obtaining the lyrics timestamp of the song audio according to the lyrics text corresponding to each song audio frame.

[0131] The lyrics text corresponding to the song audio frame can be obtained through the above steps, and the lyrics timestamp can be obtained by combining the lyrics text corresponding to each song audio frame. For example, the lyrics timestamp of the song audio can be determined based on the lyrics text corresponding to each song audio frame and the timing corresponding to each song audio frame.

[0132] In this embodiment, the text feature extraction module and the audio feature extraction module in the feature extraction model can be used to extract features of lyrics text and song audio frames of different modalities respectively. Since the text feature extraction module and the audio feature extraction module have been pre-trained by the model to output the same or similar feature expressions for text and audio representing the same semantics, this embodiment can efficiently and accurately determine the lyrics text corresponding to the song audio frame by comparing the similarity of lyrics text features. Compared with the traditional method of converting text into phonemes and then into a state sequence of HMM, and then performing Viterbi decoding according to the state transition probability, the present embodiment does not need to introduce an additional phoneme dictionary, but directly aligns the audio and text in the word dimension. Not only is the process more concise, but the effect is better, and the alignment accuracy of the lyrics timestamps finally obtained can be effectively improved.

[0133] In addition, manually generating lyrics timestamps has a certain threshold, which is a challenge for relevant users. The lyrics timestamp acquisition method of this embodiment can help musicians simplify the process of publishing works and lower the threshold for musicians to create. After acquiring the text features of multiple words in the text feature space, they can be reused and quickly matched to the corresponding lyrics text to generate lyrics timestamps for song audios in different language types, effectively covering the production process of musicians and attracting more users to create music.

[0134] In order to enable those skilled in the art to better understand the above steps, Figure 2 The embodiments of the present application are described by way of example only, but it should be understood that the embodiments of the present application are not limited thereto.

[0135] like Figure 2As shown, first, obtain the sample lyric text and sample song audio for training the model, align them through the lyric timestamp model to obtain multiple audio frames associated with each original lyric character, and use the multiple audio frames associated with the original lyric character as the multiple audio frames associated with the corresponding sample lyric character. Taking the original lyric character "我" as an example, through the lyric timestamp model, 6 audio frames associated with the original lyric character "我" can be determined, and the 6 audio frames are used as the 6 audio frames associated with the sample lyric character "我_起始".

[0136] Then, the pronunciation stage of the 6 audio frames associated with the original lyric character "我" can be recognized and labeled. According to the pronunciation stage recognition result, the 1st and 2nd audio frames among the 6 audio frames are labeled as the audio frames in the pronunciation start stage, the 3rd and 4th audio frames among the 6 audio frames are labeled as the audio frames in the pronunciation middle stage, and the 5th and 6th audio frames among the 6 audio frames are labeled as the audio frames in the pronunciation end stage.

[0137] For the 3rd and 4th audio frames labeled as the pronunciation middle stage, use them as the sample audio frames corresponding to the sample lyric character "我_中间", and then one of the sample audio frames can form a positive example with the sample lyric character "我_中间".

[0138] For the pronunciation start stage or the pronunciation end stage, taking the pronunciation start stage as an example for illustration, only the 1st and 2nd audio frames labeled as the pronunciation start stage can be used as the sample audio frames corresponding to the sample lyric character "我_起始"; of course, it can also be expanded. In some embodiments, the multiple audio frames associated with two consecutive original lyric characters are not consecutive. As Figure 2 shown, there are several unclassified audio frames between the 6 audio frames associated with the original lyric character "我" and the 6 audio frames associated with the original lyric character "爱". In addition to using the 1st and 2nd audio frames labeled as the pronunciation start stage as the sample audio frames corresponding to the sample lyric character "我_起始", the 1 audio frame before the audio frame labeled as the pronunciation start stage is also used as a sample audio frame. By expanding the audio frames in the pronunciation start stage and the pronunciation end stage, audio frames that may be misidentified as environmental sounds but are actually related to lyric characters can be correctly recognized, avoiding omission. Figure 2 The processing of the original lyric characters "爱" and "你" is similar to that of the original lyric character "我", and will not be elaborated here.

[0139] Then, input the sample audio frames into the audio feature extraction module to extract audio features; on the other hand, input the sample lyric characters into the text feature extraction module to extract text features, input the two-way representations into two-way fully connected layers respectively, calculate the contrast loss value at the output end of the fully connected layer, and optimize the loss function to train until the model converges to obtain the feature extraction model.

[0140] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0141] In an exemplary embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as shown in FIG. Figure 5 As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. Among them, the processor, the memory and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store audio song data and lyrics text data. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a method for obtaining lyrics timestamp is implemented.

[0142] In an exemplary embodiment, a computer device is provided. The computer device may be a terminal, and its internal structure diagram may be as shown in FIG. Figure 6As shown. The computer device includes a processor, a memory, an input / output interface, a communication interface, a display unit and an input device. The processor, the memory and the input / output interface are connected through a system bus, and the communication interface, the display unit and the input device are connected to the system bus through the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be implemented through WIFI, a mobile cellular network, near field communication (NFC) or other technologies. When the computer program is executed by the processor, a method for obtaining lyrics timestamp is implemented. The display unit of the computer device is used to form a visually visible picture, which can be a display screen, a projection device or a virtual reality imaging device. The display screen can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covering the display screen, or a button, trackball or touchpad set on the computer device shell, or an external keyboard, touchpad or mouse.

[0143] Those skilled in the art will understand that Figure 5 and Figure 6 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0144] In one embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in the above-mentioned method embodiments when executing the computer program.

[0145] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0146] In one embodiment, a computer program product is provided, including a computer program, which implements the steps in the above method embodiments when executed by a processor.

[0147] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, played data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.

[0148] A person of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment method can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, an artificial intelligence (AI) processor, etc., but are not limited to this.

[0149] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0150] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A feature extraction model training method, characterized in that: The method comprises: Obtain a sample song audio and a plurality of sample lyrics corresponding to the sample song audio; wherein the sample lyrics are obtained after annotating the pronunciation of the original lyrics in the sample lyrics text corresponding to the sample song audio; According to the pronunciation stage of each of the sample lyrics, determine the sample audio frame corresponding to each of the sample lyrics from the sample song audio; According to the plurality of sample audio frames and the sample lyrics text corresponding to each of the sample audio frames, a positive sample pair and a negative sample pair are determined; the positive sample pair includes the sample audio frame and the sample lyrics text corresponding to the sample audio frame, and the negative sample pair includes the sample audio frame and the sample lyrics text not corresponding to the sample audio frame; According to the positive sample pairs and the negative sample pairs, comparative learning training is performed on the audio feature extraction module and the text feature extraction module to obtain a trained feature extraction model; wherein the audio feature extraction module is used to extract audio features of audio frames; and the text feature extraction module is used to extract text features of lyrics.

2. The method according to claim 1, characterized in that The step of performing comparative learning training on the audio feature extraction module and the text feature extraction module according to the positive sample pair and the negative sample pair to obtain a trained feature extraction model includes: The audio feature extraction module obtains the audio feature of each of the sample audio frames in the plurality of sample audio frames, and the text feature extraction module determines the text feature of each of the sample lyrics text; Determine the contrast loss value according to the first feature difference and the second feature difference; the first feature difference is the difference between the audio feature of the sample audio frame in the positive sample pair and the text feature of the sample lyrics, and the second feature difference is the difference between the audio feature of the sample audio frame in the negative sample pair and the text feature of the sample lyrics; The module parameters of the audio feature extraction module and the text feature extraction module are adjusted according to the contrast loss value to obtain a trained feature extraction model.

3. The method according to claim 1, characterized in that The step of determining, from the sample song audio, a sample audio frame corresponding to each sample lyric text according to the pronunciation stage of each sample lyric text comprises: Determining multiple audio frames associated with each of the original lyrics texts from the sample song audio according to the lyrics timestamps corresponding to the sample lyrics texts; For each of the sample lyrics, multiple audio frames associated with the original lyrics corresponding to the sample lyrics are used as multiple audio frames associated with the sample lyrics; For each of the sample lyrics, the sample audio frame corresponding to the sample lyrics is determined from the sample song audio according to the pronunciation stage of the sample lyrics and the pronunciation stage of each audio frame in the multiple audio frames associated with the sample lyrics.

4. The method according to claim 3, characterized in that The pronunciation stage is a pronunciation start stage, a pronunciation middle stage or a pronunciation end stage; the step of determining the sample audio frame corresponding to the sample lyrics from the sample song audio according to the pronunciation stage of the sample lyrics and the pronunciation stage of each audio frame in the multiple audio frames associated with the sample lyrics includes: If the pronunciation stage of the sample lyrics is the middle stage of pronunciation, determine the audio frame whose pronunciation stage is marked as the middle stage of pronunciation among the multiple audio frames associated with the sample lyrics as the sample audio frame corresponding to the sample lyrics; If the pronunciation stage of the sample lyrics is the pronunciation start stage, determine the audio frame whose pronunciation stage is marked as the pronunciation start stage among the multiple audio frames associated with the sample lyrics as the sample audio frame corresponding to the sample lyrics; If the pronunciation stage of the sample lyrics is the pronunciation end stage, the audio frame whose pronunciation stage is marked as the pronunciation end stage among the multiple audio frames associated with the sample lyrics is determined as the sample audio frame corresponding to the sample lyrics.

5. The method according to claim 3, characterized in that: The pronunciation stage is a pronunciation start stage, a pronunciation middle stage or a pronunciation end stage; the step of determining the sample audio frame corresponding to the sample lyrics from the sample song audio according to the pronunciation stage of the sample lyrics and the pronunciation stage of each audio frame in the multiple audio frames associated with the sample lyrics includes: If the pronunciation stage of the sample lyrics is an intermediate pronunciation stage, determining the audio frame marked as the intermediate pronunciation stage among the multiple audio frames associated with the sample lyrics as the sample audio frame corresponding to the sample lyrics; If the pronunciation stage of the sample lyrics is the pronunciation start stage, the audio frames marked as the pronunciation start stage among the multiple audio frames associated with the sample lyrics and the preset number of audio frames before the audio frames in the pronunciation start stage in the sample song audio are determined as the sample audio frames corresponding to the sample lyrics; If the pronunciation stage of the sample lyrics text is the pronunciation end stage, the audio frames marked as the pronunciation end stage among the multiple audio frames associated with the sample lyrics text, and a preset number of audio frames after the audio frames in the pronunciation end stage in the sample song audio are determined as sample audio frames corresponding to the sample lyrics text.

6. The method according to claim 3, characterized in that The pronunciation stage is a pronunciation start stage, a pronunciation middle stage or a pronunciation end stage; the method further comprises: Obtaining the audio frame energy of each audio frame in a plurality of audio frames associated with the original lyrics text; The audio frame energy is greater than or equal to the energy threshold and the audio frame in the middle of the multiple audio frames is marked as an audio frame in the middle stage of pronunciation; Marking the audio frame whose energy is less than the energy threshold and before the audio frame in the middle stage of pronunciation among the multiple audio frames as an audio frame in the initial stage of pronunciation; The audio frame energy of the audio frame is less than the energy threshold and the audio frame after the audio frame in the middle stage of pronunciation among the multiple audio frames is marked as an audio frame in the end stage of pronunciation.

7. A method for obtaining lyrics timestamp, characterized in that: The method comprises: Obtain song audio and lyrics text corresponding to the song audio, wherein the song audio includes a plurality of song audio frames, and the lyrics text includes a plurality of lyrics characters; The song audio is input into a trained feature extraction model, and an audio feature extraction module in the feature extraction model performs feature extraction on at least part of the multiple song audio frames to obtain lyrics text features corresponding to each of the at least part of the song audio frames, wherein the feature extraction model is trained by the feature extraction model training method according to any one of claims 1 to 6; Obtaining lyrics text features corresponding to each of the multiple lyrics texts from a pre-constructed feature dictionary; the lyrics text features in the feature dictionary are extracted by a text feature extraction module in the feature extraction model; Determine the lyrics text corresponding to each of the song audio frames according to the similarity between the lyrics text features corresponding to each of the song audio frames and the lyrics text features corresponding to each of the lyrics texts; The lyrics timestamp of the song audio is obtained according to the lyrics text corresponding to each of the song audio frames.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the feature extraction model training method described in any one of claims 1 to 6 or the steps of the lyrics timestamp acquisition method described in claim 7 are implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the feature extraction model training method described in any one of claims 1 to 6 or the steps of the lyrics timestamp acquisition method described in claim 7 are implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the feature extraction model training method described in any one of claims 1 to 6 or the steps of the lyrics timestamp acquisition method described in claim 7 are implemented.

Images