Lyrics acoustic model training method, lyrics recognition method, device and product

Abstract

The present application relates to the field of audio processing technology, and provides a training method for a lyrics acoustic model, a lyrics recognition method, a computer device, and a computer program product, which can effectively improve the recognition performance of the lyrics acoustic model. The training method of the lyrics acoustic model includes: obtaining a song sample and the phoneme label of the lyrics in the song sample; obtaining the song spectrum information of the song sample and the accompaniment spectrum information of the song sample; inputting the song spectrum information and the accompaniment spectrum information into the lyrics acoustic model to be trained, and extracting the song features of the song spectrum information and the accompaniment features of the accompaniment spectrum information by the lyrics acoustic model to be trained, and outputting the predicted lyrics phonemes according to the song features and the accompaniment features; determining the model loss according to the predicted lyrics phonemes and phoneme labels; adjusting the model parameters of the lyrics acoustic model to be trained according to the model loss until the training end condition is met to obtain a trained lyrics acoustic model.

Description

Technical Field

[0001] The present application relates to the field of audio processing technology, and in particular to a lyrics acoustic model training method, a lyrics recognition method, a computer device and a computer program product. Background Art

[0002] With the development of computer technology, the lyrics text in the song audio can be quickly extracted through singing voice recognition technology. In traditional technology, the human voice can be extracted by eliminating the accompaniment in the song audio and an acoustic model can be trained based on the extracted human voice. In the subsequent singing voice recognition process, the acoustic model can be used to obtain the lyrics phonemes in the song audio and determine the corresponding lyrics text.

[0003] However, in the process of eliminating the accompaniment, part of the vocals in the song are easily eliminated, which affects the reliability of the vocals used to train the acoustic model and limits the performance of the acoustic model. Summary of the invention

[0004] Based on this, it is necessary to provide a lyrics acoustic model training method, lyrics recognition method, computer equipment and computer program product to address the above technical problems.

[0005] In a first aspect, the present application provides a method for training a lyrics acoustic model. The method comprises:

[0006] Obtaining song samples and phoneme labels of lyrics in the song samples;

[0007] Acquire song spectrum information of the song sample and accompaniment spectrum information of the song sample;

[0008] Inputting the song spectrum information and the accompaniment spectrum information into a lyrics acoustic model to be trained, extracting song features of the song spectrum information and accompaniment features of the accompaniment spectrum information by the lyrics acoustic model to be trained, and outputting predicted lyrics phonemes according to the song features and the accompaniment features;

[0009] Determining a model loss based on the predicted lyrics phonemes and the phoneme labels;

[0010] The model parameters of the lyrics acoustic model to be trained are adjusted according to the model loss until the training end condition is met to obtain a trained lyrics acoustic model.

[0011] In a second aspect, the present application also provides a lyrics recognition method. The method comprises:

[0012] Acquire the song spectrum information of the song whose lyrics are to be identified and the accompaniment spectrum information of the song;

[0013] Inputting the song spectrum information and the accompaniment spectrum information of the song into the lyrics acoustic model to obtain the predicted lyrics phonemes of the song output by the lyrics acoustic model; the lyrics acoustic model is trained according to the training method of the lyrics acoustic model described in any one of the above items;

[0014] The predicted lyrics phonemes of the song are input into a language model to obtain the lyrics of the song output by the language model.

[0015] In a third aspect, the present application further provides a computer device. The computer device includes a memory and a processor, the memory stores a computer program, and the processor implements the following steps when executing the computer program:

[0016] Obtaining song samples and phoneme labels of lyrics in the song samples;

[0017] Acquire song spectrum information of the song sample and accompaniment spectrum information of the song sample;

[0018] Inputting the song spectrum information and the accompaniment spectrum information into a lyrics acoustic model to be trained, extracting song features of the song spectrum information and accompaniment features of the accompaniment spectrum information by the lyrics acoustic model to be trained, and outputting predicted lyrics phonemes according to the song features and the accompaniment features;

[0019] Determining a model loss based on the predicted lyrics phonemes and the phoneme labels;

[0020] The model parameters of the lyrics acoustic model to be trained are adjusted according to the model loss until the training end condition is met to obtain a trained lyrics acoustic model.

[0021] In a fourth aspect, the present application further provides a computer device. The computer device includes a memory and a processor, the memory stores a computer program, and the processor implements the following steps when executing the computer program:

[0022] Acquire the song spectrum information of the song whose lyrics are to be identified and the accompaniment spectrum information of the song;

[0023] Inputting the song spectrum information and the accompaniment spectrum information of the song into the lyrics acoustic model to obtain the predicted lyrics phonemes of the song output by the lyrics acoustic model; the lyrics acoustic model is trained according to the training method of the lyrics acoustic model described in any one of the above items;

[0024] The predicted lyrics phonemes of the song are input into a language model to obtain the lyrics of the song output by the language model.

[0025] In a fifth aspect, the present application further provides a computer program product. The computer program product includes a computer program, and when the computer program is executed by a processor, the following steps are implemented:

[0026] Obtaining song samples and phoneme labels of lyrics in the song samples;

[0027] Acquire song spectrum information of the song sample and accompaniment spectrum information of the song sample;

[0028] Inputting the song spectrum information and the accompaniment spectrum information into a lyrics acoustic model to be trained, extracting song features of the song spectrum information and accompaniment features of the accompaniment spectrum information by the lyrics acoustic model to be trained, and outputting predicted lyrics phonemes according to the song features and the accompaniment features;

[0029] Determining a model loss based on the predicted lyrics phonemes and the phoneme labels;

[0030] The model parameters of the lyrics acoustic model to be trained are adjusted according to the model loss until the training end condition is met to obtain a trained lyrics acoustic model.

[0031] In a sixth aspect, the present application further provides a computer program product. The computer program product includes a computer program, and when the computer program is executed by a processor, the following steps are implemented:

[0032] Acquire the song spectrum information of the song whose lyrics are to be identified and the accompaniment spectrum information of the song;

[0033] Inputting the song spectrum information and the accompaniment spectrum information of the song into the lyrics acoustic model to obtain the predicted lyrics phonemes of the song output by the lyrics acoustic model; the lyrics acoustic model is trained according to the training method of the lyrics acoustic model described in any one of the above items;

[0034] The predicted lyrics phonemes of the song are input into a language model to obtain the lyrics of the song output by the language model.

[0035] The training method, lyrics recognition method, computer device and computer program product of the lyrics acoustic model can obtain song samples and phoneme labels of lyrics in the song samples, obtain song spectrum information of the song samples and accompaniment spectrum information of the song samples, input the song spectrum information and accompaniment spectrum information into the lyrics acoustic model to be trained, extract song features of the song spectrum information and accompaniment features of the accompaniment spectrum information by the lyrics acoustic model to be trained, output predicted lyrics phonemes according to the song features and accompaniment features, determine model loss according to the predicted lyrics phonemes and phoneme labels, adjust model parameters of the lyrics acoustic model to be trained according to the model loss, until the training end condition is met, and obtain a trained lyrics acoustic model. This embodiment can enable the lyrics acoustic model to fully learn the song features while acquiring the accompaniment characteristics of the accompaniment by acquiring the song features and accompaniment features of the sample song respectively, effectively eliminate the interference caused by the accompaniment in the lyrics phoneme prediction process, and improve the recognition performance of the lyrics acoustic model. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 A schematic diagram of a flow chart of a method for training a lyrics acoustic model in one embodiment;

[0037] Figure 2 A schematic diagram of a process of predicting lyrics phonemes in one embodiment;

[0038] Figure 3 A schematic diagram of a process of outputting phoneme probability distribution in one embodiment;

[0039] Figure 4 A diagram of an application environment of a lyrics recognition method in an embodiment;

[0040] Figure 5 A schematic diagram of a process flow of a lyrics recognition method in one embodiment;

[0041] Figure 6 A schematic diagram of a process for obtaining a singing voice recognition model in one embodiment;

[0042] Figure 7 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0043] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0044] With the development of computer technology, the lyrics text in the song audio can be quickly extracted through singing voice recognition technology. In traditional technology, the human voice can be extracted by eliminating the accompaniment in the song, and an acoustic model can be trained based on the extracted human voice. In the subsequent singing voice recognition process, the acoustic model is used to obtain the lyrics phonemes in the song and determine the corresponding lyrics text.

[0045] However, in the above method, although the former can eliminate the interference of the accompaniment on the model, it is easy to eliminate part of the human voice in the song in the process of eliminating the accompaniment, and it is difficult to extract the complete human voice from the song. For example, the human voice in the case of bubble sound, breath, long vowel tail, etc. will be mistakenly eliminated, thereby affecting the reliability of the human voice used to train the acoustic model, and the acoustic model obtained only based on the human voice training is not robust, and the performance of the acoustic model is limited. In order to at least solve the above technical problems, the present application provides a method for training a lyrics acoustic model.

[0046] In one embodiment, Figure 1 As shown, a method for training a lyrics acoustic model is provided. This embodiment uses the method applied to a terminal as an example. It can be understood that the method can also be applied to a server, and can also be applied to a system including a terminal and a server, and is implemented through the interaction between the terminal and the server. In this embodiment, the method includes the following steps:

[0047] S101, obtaining song samples and phoneme labels of lyrics in the song samples.

[0048] Among them, phonemes are the smallest units of speech. In practice, corresponding phonemes can be identified based on the pronunciation actions in a syllable, and one pronunciation action constitutes one phoneme. As an example, a phoneme tag can indicate the phonemes of lyrics in a song sample, and the tone of the phoneme can be the pronunciation tone of the lyrics in the song sample.

[0049] In practical applications, song samples for training lyrics acoustic models can be obtained, and phoneme labels of lyrics in the song samples can be obtained.

[0050] S102, obtaining song frequency spectrum information of the song sample and accompaniment frequency spectrum information of the song sample.

[0051] The spectrum information may be information reflecting spectrum characteristics. For the sake of distinction, the spectrum information of a song sample is referred to as song spectrum information, and the spectrum information of the accompaniment of the song sample is referred to as accompaniment spectrum information.

[0052] After obtaining the song sample, the song spectrum information of the song sample and the accompaniment spectrum information of the song sample can be obtained. For example, for the obtained song sample, the accompaniment can be extracted from the song sample by a preset separation model, such as using a spleeter separation model or other separation models to extract the accompaniment audio, and then the corresponding Mel Frequency Cepstrum Coefficient (MFCC) can be obtained based on the audio of the song sample and the audio of the accompaniment, as the song spectrum information of the song sample and the accompaniment spectrum information.

[0053] S103, inputting the song spectrum information and the accompaniment spectrum information into the lyrics acoustic model to be trained, extracting the song features of the song spectrum information and the accompaniment features of the accompaniment spectrum information by the lyrics acoustic model to be trained, and outputting the predicted lyrics phonemes according to the song features and the accompaniment features.

[0054] After obtaining the song spectrum information and the accompaniment spectrum information, the song spectrum information and the accompaniment spectrum information can be input into the lyrics acoustic model to be trained. Specifically, the song spectrum information can cover a variety of rich semantic information, such as the lyrics information, singer information and accompaniment information of the song. In this embodiment, on the one hand, the lyrics acoustic model to be trained can learn various types of semantic information from the song spectrum information to obtain song features; on the other hand, the lyrics acoustic model to be trained can specifically learn the characteristics of the accompaniment spectrum information to obtain accompaniment features, thereby learning the song features and accompaniment features separately, and the two do not interfere with each other.

[0055] Among them, by learning the accompaniment characteristics of the accompaniment spectrum information, the lyrics acoustic model can learn the interference information existing in the process of lyrics phoneme prediction, and eliminate the accompaniment interference contained in the song characteristics according to the accompaniment characteristics, without affecting the song characteristics related to the human voice, ensuring that the human voice information is fully retained, and then predict the lyrics phonemes on this basis to improve the accuracy of the final output lyrics phonemes.

[0056] In some acoustic model training methods, in order to retain complete vocal information, the accompaniment and vocals in the song may not be separated, but the complete song may be used directly to train the deep neural network. Through model training, the deep neural network can strip off information irrelevant to the target (i.e., the lyrics phonemes), eliminating the impact of the accompaniment in the song during the lyrics phoneme recognition process. Although deep neural networks have a certain degree of noise resistance, and the label-based learning process can strip off information irrelevant to the target, the more irrelevant information there is, the more difficult it is to train the model. In this method, not doing any processing on the accompaniment will significantly increase the difficulty of model training.

[0057] The lyrics acoustic model training method of the present embodiment can, while extracting song features from the song spectrum information containing the complete semantic information of the song, additionally perform diversion learning on the accompaniment, so that the model can separately learn song features covering multiple semantic information of the song (including the accompaniment information of the song), as well as the accompaniment features corresponding to the interfering accompaniment, thereby providing a basis for the subsequent elimination of accompaniment interference that is not related to the human voice in the song; and different types of songs have obvious differences in accompaniment, and by specifically learning the features of the interfering factors (i.e., the accompaniment features) in the process of lyric phoneme prediction, the difficulty of network training can be significantly reduced.

[0058] S104, determining the model loss according to the predicted lyric phonemes and phoneme labels.

[0059] S105, adjusting the model parameters of the lyrics acoustic model to be trained according to the model loss until the training end condition is met, thereby obtaining a trained lyrics acoustic model.

[0060] Specifically, after obtaining the predicted lyrics phonemes output by the lyrics acoustic model to be trained, the model loss can be determined based on the predicted lyrics phonemes and phoneme labels, and the model parameters of the lyrics acoustic model to be trained can be adjusted based on the model loss. For example, the model loss is determined based on the difference between the predicted lyrics phonemes and phoneme labels, and the model parameters of the lyrics acoustic model to be trained are adjusted through the back propagation algorithm. Iterate multiple times until the training end condition is met, and the trained lyrics acoustic model can be obtained.

[0061] In this embodiment, a song sample and the phoneme labels of the lyrics in the song sample can be obtained, the song spectrum information of the song sample and the accompaniment spectrum information of the song sample can be obtained, the song spectrum information and the accompaniment spectrum information can be input into the lyrics acoustic model to be trained, the lyrics acoustic model to be trained takes the song features of the song spectrum information and extracts the accompaniment features of the accompaniment spectrum information, outputs the predicted lyrics phonemes according to the song features and the accompaniment features, determines the model loss according to the predicted lyrics phonemes and the phoneme labels, and adjusts the model parameters of the lyrics acoustic model to be trained according to the model loss until the training end condition is met, thereby obtaining a trained lyrics acoustic model. By respectively obtaining the song features and accompaniment features of the sample song, this embodiment enables the lyrics acoustic model to obtain the accompaniment characteristics of the accompaniment while fully learning the song features, effectively eliminating the interference of the accompaniment in the song on the prediction of lyrics phonemes, and improving the recognition performance of the lyrics acoustic model.

[0062] In one embodiment, the song spectrum information may include song frame spectrum information of each song frame of the song sample; for example, the audio of the song sample may be pre-processed by pre-emphasis, framing, windowing, etc. to improve the speech signal-to-noise ratio, and then the time domain audio data of each song frame obtained after the preprocessing is converted into frequency domain data by short-time Fourier transform (STFT, short-timeFourier transform, or short-term Fourier transform), and the obtained frequency domain is input into a Mel filter for filtering to obtain a Mel spectrum reflecting the auditory reception frequency of the human ear, and then the logarithm of the Mel frequency is taken and discrete cosine transform (DCT for Discrete Cosine Transform) is performed to obtain the accompaniment frame spectrum information of each song frame.

[0063] The lyrics acoustic model to be trained may include a first network for extracting song features, and the first network may include multiple hidden layers. Exemplarily, the first network may be a time-delay neural network.

[0064] S103 extracts song features of song spectrum information from the lyrics acoustic model to be trained, which may include the following steps:

[0065] The song frame spectrum information of each song frame is input into the first network, and the first network fuses the input features in the input feature sequence according to the input feature sequence and delay value provided by the previous hidden layer to obtain the input feature sequence of the current hidden layer; the song features of the song frame spectrum information of each song frame are obtained according to the output result of the last hidden layer.

[0066] Among them, the input feature sequence of the first hidden layer is the song frame spectrum information of each song frame.

[0067] In practical applications, after the song frame spectrum information of each song frame is input into the first network, the first network can fuse the input features in the input feature sequence according to the input features of the previous hidden layer and the delay value of the previous hidden layer.

[0068] Specifically, for each input feature in the input feature sequence provided by the previous hidden layer, multiple input features adjacent to the input feature can be determined according to the delay value, and feature fusion can be performed on the input feature and its adjacent multiple input features. For example, the input feature, the first n input features of the input feature, and the m input features after the input feature are fused to obtain a fused feature associated with the input feature and its adjacent multiple input features. The above processing is performed on each input feature in the input feature sequence provided by the previous hidden layer in turn to obtain a fused feature corresponding to each input feature, and then the input feature sequence of the current hidden layer can be obtained based on multiple fused features. This network structure is conducive to learning the contextual relevance of time series signals. As a time series signal with strong before-and-after relevance, songs can fully learn the relevance of semantic information of previous and next songs through the first network, especially the relevance of lyrics, to ensure the consistency of recognition results.

[0069] For the first hidden layer in the first network, the song spectrum information of each song frame can be used as an input feature sequence. Therefore, in the first hidden layer, for each song frame spectrum information, feature fusion can be performed on the song frame spectrum information and the song frame spectrum information of several song frames before and after it according to the delay value.

[0070] The processing is performed in turn through each hidden layer of the first network, and then the song features of the song frame spectrum information of each song frame can be obtained according to the output result of the last hidden layer.

[0071] In this embodiment, the first network can fuse the input features in the input feature sequence according to the input feature sequence and delay value provided by the previous hidden layer, and associate the input features of multiple adjacent moments in each hidden layer, so as to effectively identify the correlation between the song voice information and extract more reliable and accurate song features.

[0072] In one embodiment, the accompaniment spectrum information may include the accompaniment frame spectrum information of each accompaniment frame of the song sample. For example, after extracting the accompaniment audio from the audio of the song sample, the accompaniment audio may be pre-processed by pre-emphasis, framing, windowing, etc., and then the time domain audio data of each accompaniment frame obtained after the preprocessing is converted into frequency domain data by short-time Fourier transform, and the obtained frequency domain is input into a Mel filter for filtering to obtain a Mel spectrum, and then the logarithm of the Mel frequency therein is taken and discrete cosine transform is performed to obtain the accompaniment frame spectrum information of each accompaniment frame.

[0073] The lyrics acoustic model to be trained may include a second network for extracting accompaniment features, and the second network may include multiple residual blocks. Exemplarily, the second network may be a residual neural network (Resnet), which is a convolutional neural network with a residual block added thereto. Specifically, the second network may include multiple residual blocks, wherein each residual block may be composed of a direct mapping part and a residual part, and the residual part may include at least one convolutional layer. For the residual block input x i , we can use this input x l They are input into the direct mapping part and the residual part for processing respectively. The residual part can process the input x l Perform convolution operation to get F(x), and the direct mapping part can convert x l Directly output, and then the convolution operation result F(x) of the residual part can be combined with the output x of the direct mapping part l , we get F(x)+x l The structure of the residual block can introduce the input of the previous layer into the current layer, avoiding the problem of gradient vanishing or gradient exploding during the deepening of the network, and has stronger learning ability than ordinary convolutional neural networks.

[0074] S103 extracts accompaniment features of accompaniment spectrum information from the lyrics acoustic model to be trained, which may include:

[0075] The accompaniment spectrum information of each accompaniment frame is input into the second network, and the second network sequentially transfers the features of the accompaniment frame spectrum information of each accompaniment frame from the first residual block to the last residual block to obtain the accompaniment features of the accompaniment frame spectrum information of each accompaniment frame.

[0076] In this step, after obtaining the accompaniment spectrum information of each accompaniment frame, the accompaniment spectrum information of each accompaniment frame can be input into the second network, and the accompaniment frame spectrum information of each accompaniment frame can be transferred from the first residual block to the last residual block by multiple residual blocks in the second network in turn, and then the accompaniment features of the accompaniment frame spectrum information of each accompaniment frame can be obtained according to the output of the second network.

[0077] In this embodiment, the accompaniment spectrum information of the accompaniment frame is processed by multiple residual blocks in the second network, which can avoid gradient vanishing or gradient explosion when the detail information in the accompaniment spectrum information is effectively captured by the deep neural network, thereby providing a basis for the model to more accurately identify the interference information in the song features.

[0078] In one embodiment, the song spectrum information includes the song frame spectrum information of each song frame of the song sample, and the accompaniment spectrum information includes the accompaniment frame spectrum information of each accompaniment frame of the song sample, such as Figure 2As shown, outputting predicted lyrics phonemes according to song features and accompaniment features in S103 may include the following steps:

[0079] S201, obtaining song features of song frame frequency spectrum information of each song frame and accompaniment features of accompaniment frame frequency spectrum information of each accompaniment frame.

[0080] After inputting each song frame spectrum information and each accompaniment frame spectrum information into the first network and the second network respectively, the song features output by the first network for each song frame spectrum information and the accompaniment features output by the second network for each accompaniment frame spectrum information can be obtained.

[0081] S202, concatenating the song feature of each song frame and the accompaniment feature of each corresponding accompaniment frame, and inputting each concatenated song feature and accompaniment feature into the classifier of the lyrics acoustic model to be trained.

[0082] In practical applications, the accompaniment can be extracted from the song sample, and accordingly, the song frame and the accompaniment frame can have a corresponding relationship. For the song features of each song frame output by the first network, the song features of the song frame can be fused with the accompaniment features of the corresponding accompaniment frame, and the song features and the accompaniment features can be spliced, and each spliced ​​song feature and accompaniment feature can be input into the classifier of the lyrics acoustic model to be trained.

[0083] In some examples, when fusing song features and accompaniment features, they can be concatenated by adding, concatenating, or subtracting. For example, the song features and their corresponding accompaniment features can be summed. In some related feature fusion methods, two feature vectors can be combined into a complex vector and the complex vector can be used as the feature obtained after summing (for example, for input features x and y, z=x+iy, where i is an imaginary unit). For another example, the song features and their corresponding accompaniment features can be subtracted, and the features obtained after subtracting can be input into a classifier.

[0084] S203, obtaining predicted lyrics phonemes of each song frame according to the output result of the classifier.

[0085] After the concatenated song features and accompaniment features are input into the classifier of the lyrics acoustic model to be trained, the classifier can perform multi-classification processing on each concatenated song feature and accompaniment feature, and then the predicted lyrics phoneme of each song frame can be obtained according to the output result of the classifier. Exemplarily, the classifier in the lyrics acoustic model to be trained can be a Softmax layer, and the predicted lyrics phoneme of each frame can be a phoneme probability distribution, or can also be a specific phoneme.

[0086] In this embodiment, the song features of the song frame spectrum information of each song frame and the accompaniment features of the accompaniment frame spectrum information of each accompaniment frame can be obtained, the song features of each song frame and the accompaniment features of each corresponding accompaniment frame are spliced, and each spliced ​​song feature and accompaniment feature is input into the classifier of the lyrics acoustic model to be trained, and then the predicted lyrics phonemes of each song frame can be obtained according to the output results of the classifier, so that the lyrics phonemes of the song can be accurately identified at the frame level.

[0087] In one embodiment, obtaining the predicted lyric phonemes of each song frame based on the output results of the classifier may include the following steps: obtaining the phoneme probability distribution of each song frame output by the classifier; and determining the predicted lyric phonemes of each song frame based on the phoneme probability distribution of each song frame.

[0088] As an example, the phoneme probability distribution can indicate the probability that the lyrics phoneme of the song frame belongs to each pre-provided phoneme. In a specific implementation, after each spliced ​​song feature and accompaniment feature is input into the classifier, the classifier can identify the features that exist as interference parts in the song features based on the accompaniment features, that is, the accompaniment features covered by the song features. Specifically, in the process of lyrics phoneme recognition, the accompaniment in the song often has a greater impact on the accuracy of lyrics phoneme recognition. In this embodiment, the song features extracted by the lyrics acoustic model contain a variety of semantic information of the song. Accordingly, the song features also cover the accompaniment features associated with the accompaniment part. After obtaining the spliced ​​song features and accompaniment features, the classifier can use the input accompaniment features as a reference to determine the accompaniment features associated with the accompaniment part in the song features, thereby identifying the interference information in the song features. After the classifier identifies the interference information in the song features, it can make predictions based on other information in the song features except the interference information to obtain the phoneme probability distribution of the song frame. For example, Figure 3 As shown, n phonemes can be provided in advance, and the classifier can determine the posterior probability p1, p2...pn that the phonemes in the song frame belong to each phoneme based on the concatenated one-dimensional features, namely the song features and the accompaniment features, and obtain the phoneme probability distribution.

[0089] Furthermore, the lyric phonemes of each song frame can be predicted based on the phoneme probability distribution of each song frame. For example, after obtaining the phoneme probability distribution, the phoneme corresponding to the maximum probability can be determined as the lyric phoneme of the song frame.

[0090] In this embodiment, the phoneme probability distribution of each song frame output by the classifier can be obtained, so that the predicted lyrics phonemes of each song frame can be determined based on the phoneme probability distribution of each song frame, and reliable predicted lyrics phonemes can be screened out by comparing the probabilities of song frames belonging to different phonemes.

[0091] In one embodiment, in S104, determining the model loss according to the predicted lyric phonemes and phoneme labels may include: obtaining the phoneme labels of the lyrics of each song frame according to the phoneme labels of the lyrics in the song sample; determining the model loss according to the predicted lyric phonemes of each song frame and the phoneme labels of the lyrics of each song frame.

[0092] In practical applications, the phoneme labels of the lyrics in the song sample can, in addition to indicating the phonemes of the lyrics, also indicate the time when the phonemes of the lyrics appear in the song sample, that is, the song frames corresponding to the phonemes. For example, the phoneme label of the lyric "ah" can indicate that the phoneme at the song frame at time t1 is ā. In a specific implementation, the pronunciation of the lyrics of the song sample consists of multiple phonemes. When obtaining the phoneme labels, further time division can be performed on the multiple phonemes of the song sample to obtain the phoneme labels corresponding to each song frame.

[0093] After obtaining the predicted lyric phonemes output by the lyric acoustic model, the frame-level phoneme labels can be obtained according to the phoneme labels of the lyrics in the song sample, that is, the phoneme labels of the lyrics of each song frame can be obtained. Furthermore, the model loss can be determined according to the predicted lyric phonemes of each song frame and the phoneme labels of the lyrics of each song frame. Specifically, for example, the difference between the lyric phonemes of each song frame and the phoneme labels of the corresponding song frame lyrics can be obtained, and the differences of multiple song frames can be integrated to obtain the model loss. In other words, when the phoneme labels indicate the lyric phonemes at each moment (i.e., the moment corresponding to the song frame) of the song sample, the difference between the actual lyric phonemes and the predicted lyric phonemes of the song frame at that moment can be determined according to the difference between the predicted lyric phonemes and the phoneme labels at the same moment (corresponding to the same song frame), and then the differences of multiple song frames can be combined to obtain the model loss.

[0094] In this embodiment, the phoneme labels of the lyrics of each song frame can be obtained according to the phoneme labels of the lyrics in the song sample, and the model loss can be determined according to the predicted lyric phonemes of each song frame and the phoneme labels of the lyrics of each song frame, which can perform fine-grained difference comparison at the frame level and effectively improve the recognition performance of the lyric acoustic model.

[0095] In one embodiment of the present application, a lyric recognition method is further provided, which can be applied to an application environment as Figure 4 shown. In this application environment, a terminal and a server may be included, and the terminal can communicate with the server through a network.

[0096] The server may be implemented as an independent server or a server cluster composed of multiple servers. The server may have a data storage system. The data storage system may store songs with lyrics to be recognized. The data storage system may be integrated in the server or placed on a cloud or other network server. The terminal may be, but is not limited to, various personal computers, laptops, smart phones, tablet computers, and portable wearable devices.

[0097] In one embodiment, Figure 5 As shown, a lyrics recognition method is provided, which is applied to Figure 4 The server in the example is used to illustrate the following steps:

[0098] S501, obtaining the song spectrum information of the song whose lyrics are to be identified and the song's accompaniment spectrum information.

[0099] In actual applications, the terminal can send a song with lyrics to be identified to the server. After receiving the song, the server can obtain the song spectrum information and the song accompaniment spectrum information. Specifically, after obtaining the song with lyrics to be identified, the song accompaniment can be extracted from the song through a preset separation model, and the song spectrum information and the accompaniment spectrum information of the song accompaniment can be obtained respectively.

[0100] Exemplarily, the songs whose lyrics are to be identified may include at least one of the following: tracks in a song database that are not associated with lyrics, songs to be identified hummed by users, cover songs, and audio to be reviewed.

[0101] Specifically, for tracks that are not associated with lyrics in the song database, the audio of the track can be sent to the server, which will efficiently and quickly identify the lyrics of the track, reducing the cost of manual transcription; for songs to be identified that are hummed by users, the recognition accuracy and speed of the songs to be identified can be effectively improved by predicting the lyrics in the song and matching them with the lyrics in the song database; and by identifying the content of the lyrics in cover songs and the position of the lyrics in the cover songs, it is possible to assist in determining the similarity between the cover songs and the reference songs while using the song melody for identification; the audio to be reviewed can be audio collected during music security monitoring, such as audio or video uploaded by users through a terminal. By performing lyrics recognition on the audio to be reviewed, it is possible to assist in determining whether there are illegal sensitive words or sensitive word positions in the audio to be reviewed, thereby realizing the detection of the legality of the audio.

[0102] S502, inputting the song spectrum information and the song accompaniment spectrum information into the lyrics acoustic model to obtain the predicted lyrics phonemes of the song output by the lyrics acoustic model; the lyrics acoustic model is trained according to any of the above lyrics acoustic model training methods.

[0103] After obtaining the song spectrum information and the song accompaniment spectrum information of the song, the song spectrum information and the song accompaniment spectrum information of the song can be input into the lyrics acoustic model to obtain the predicted lyrics phonemes of the song output by the lyrics acoustic model.

[0104] S503, inputting the predicted lyrics phonemes of the song into the language model to obtain the lyrics of the song output by the language model.

[0105] As an example, the speech model may predict a corresponding text sequence based on an input phoneme sequence, wherein the phoneme sequence may include at least one predicted lyric phoneme.

[0106] After obtaining the predicted lyrics phonemes of the song, the predicted lyrics phonemes can be input into the language model to obtain the lyrics of the song output by the language model. In practical applications, the final recognition result of the lyrics can be obtained through the Viterbi decoding algorithm.

[0107] In this embodiment, the song spectrum information and the accompaniment spectrum information of the song whose lyrics are to be identified can be obtained, and the song spectrum information and the accompaniment spectrum information of the song can be input into the lyrics acoustic model to obtain the predicted lyrics phonemes of the song output by the lyrics acoustic model, wherein the lyrics acoustic model is trained according to any of the training methods for the lyrics acoustic model, and then the predicted lyrics phonemes can be input into the language model to obtain the lyrics of the song output by the language model. In this embodiment, the lyrics acoustic model can effectively eliminate the influence of the accompaniment, accurately identify the lyrics phonemes in the song, and then efficiently and accurately identify the lyrics of the song.

[0108] In one embodiment, the method may further include the following steps:

[0109] S601, obtaining a plurality of lyrics texts from a preset song database, and obtaining the lyrics phonemes of each lyrics text character in the lyrics text.

[0110] In a specific implementation, multiple lyrics texts can be obtained from a preset song database, wherein the language of the lyrics text can be determined according to actual conditions. After obtaining multiple lyrics texts, each lyrics text can be segmented and word segmented to obtain lyrics text characters, and lyrics phonemes of lyrics text characters can be obtained to obtain a lyrics dictionary.

[0111] S602: Determine the associated characters of each lyric text character and the occurrence probability of the associated characters based on the plurality of lyric texts.

[0112] As an example, the associated characters of a lyric text character may be characters that appear within a preset range of the lyric text character. For example, for each lyric text character in the lyric text, the m characters before and after the lyric text character may be used as the associated characters of the lyric text character.

[0113] In actual applications, after obtaining multiple lyrics texts, the lyrics text characters can be counted to determine the associated characters of each lyrics text character, and the occurrence probability of the associated characters can be determined by combining multiple lyrics texts.

[0114] S603, acquiring the lyrics phonemes of the associated characters from the lyrics phonemes of each lyrics text character in the lyrics text.

[0115] Furthermore, since the associated characters of the lyrics text characters are also recorded in the lyrics text, the lyrics phonemes of the associated characters can be acquired from the lyrics phonemes of the acquired lyrics text characters.

[0116] S604: construct a language model based on the lyrics phonemes of each lyric text character, the lyrics phonemes of the associated characters of each lyric text character, and the occurrence probability of the associated characters. The language model can be a statistical model that calculates the probability distribution of character sequences (such as phrases, sentences, paragraphs).

[0117] In this embodiment, multiple lyric texts can be obtained from a preset song database, and the lyric phonemes of each lyric text character in the lyric text can be obtained. Based on the multiple lyric texts, the associated characters of each lyric text character and the probability of occurrence of the associated characters are determined, and then the lyric phonemes of the associated characters can be obtained from the lyric phonemes of each lyric text character in the lyric text, and based on the lyric phonemes of each lyric text character, the lyric phonemes of the associated characters of each lyric text character and the probability of occurrence, a language model is constructed to provide a basis for identifying the lyric text corresponding to the predicted lyric phonemes.

[0118] In order to enable those skilled in the art to better understand the above steps, the embodiment of the present application is illustrated below by using an example, but it should be understood that the embodiment of the present application is not limited to this.

[0119] like Figure 6 As shown, song samples and lyrics text can be obtained from a song database, and the lyrics acoustic model and language model can be trained respectively. Then, the lyrics acoustic model and language model can be combined to construct a singing recognition model for converting song audio into corresponding lyrics text.

[0120] For the language model, the lyrics text can be segmented and molecularly processed to obtain multiple lyrics text characters, and the lyrics phonemes of each lyrics text character can be obtained. Moreover, for each lyrics text character, the occurrence probability of the m associated characters before and after it can be counted, and then the n-gram language model can be constructed based on the lyrics phonemes of each lyrics text character, the lyrics phonemes of the associated characters of each lyrics text character and the occurrence probability.

[0121] For the training of the lyrics acoustic model, the background accompaniment can be extracted from the song audio through the preset separation model "spleeter network", and the MFCC features of the song audio and the background accompaniment can be obtained respectively. The MFCC features of the song audio can be input into the TDNN network (Time Delay Neural Network) as the first network to obtain the corresponding song features, and the MFCC features of the background accompaniment can be input into the Resnet network (residual neural network) as the second network to obtain the corresponding accompaniment features. After the song features and the accompaniment features are concatenated, they can be input into the softmax layer of the lyrics acoustic model for classification, and the softmax layer outputs the posterior probability that each song frame belongs to each phoneme p1, p2...pn.

[0122] After the lyrics acoustic model is trained, the singing voice recognition model can be constructed by combining the n-gram language model and the lyrics acoustic model.

[0123] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0124] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Figure 7As shown. The computer device includes a processor, a memory and a network interface connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store song data. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a training method for a lyrics acoustic model or a lyrics recognition method is implemented.

[0125] Those skilled in the art will understand that Figure 7 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0126] In one embodiment, a computer device is further provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in the above method embodiments when executing the computer program.

[0127] In one embodiment, a computer program product is provided, including a computer program, which implements the steps in the above method embodiments when executed by a processor.

[0128] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties.

[0129] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but are not limited to this.

[0130] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0131] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A method for training a lyrics acoustic model, characterized in that: The method comprises: Obtaining song samples and phoneme labels of lyrics in the song samples; Acquire song spectrum information of the song sample and accompaniment spectrum information of the song sample; Inputting the song spectrum information and the accompaniment spectrum information into a lyrics acoustic model to be trained, extracting song features of the song spectrum information and accompaniment features of the accompaniment spectrum information by the lyrics acoustic model to be trained, and outputting predicted lyrics phonemes according to the song features and the accompaniment features; Determining a model loss based on the predicted lyrics phonemes and the phoneme labels; The model parameters of the lyrics acoustic model to be trained are adjusted according to the model loss until the training end condition is met to obtain a trained lyrics acoustic model.

2. The method according to claim 1, characterized in that The song spectrum information includes song frame spectrum information of each song frame of the song sample, and the lyrics acoustic model to be trained includes a first network, and the first network includes a plurality of hidden layers; The extracting song features of the song spectrum information from the lyrics acoustic model to be trained includes: The song frame spectrum information of each song frame is input into the first network, and the first network fuses the input features in the input feature sequence according to the input feature sequence and the delay value provided by the previous hidden layer to obtain the input feature sequence of the current hidden layer; wherein the input feature sequence of the first hidden layer is the song frame spectrum information of each song frame; The song features of the song frame spectrum information of each song frame are obtained according to the output result of the last hidden layer.

3. The method according to claim 1, characterized in that The accompaniment spectrum information includes accompaniment frame spectrum information of each accompaniment frame of the song sample, and the lyrics acoustic model to be trained includes a second network, and the second network includes a plurality of residual blocks; Extracting the accompaniment features of the accompaniment spectrum information from the lyrics acoustic model to be trained includes: The accompaniment spectrum information of each accompaniment frame is input into the second network, and the second network transfers the features of the accompaniment frame spectrum information of each accompaniment frame from the first residual block to the last residual block to obtain the accompaniment features of the accompaniment frame spectrum information of each accompaniment frame.

4. The method according to any one of claims 1 to 3, characterized in that: The song spectrum information includes song frame spectrum information of each song frame of the song sample, and the accompaniment spectrum information includes accompaniment frame spectrum information of each accompaniment frame of the song sample; the outputting predicted lyrics phonemes according to the song features and the accompaniment features includes: Acquire song features of song frame spectrum information of each song frame and accompaniment features of accompaniment frame spectrum information of each accompaniment frame; Splicing the song feature of each song frame and the accompaniment feature of each corresponding accompaniment frame, and inputting each spliced ​​song feature and accompaniment feature into the classifier of the lyrics acoustic model to be trained; The predicted lyrics phonemes of each song frame are obtained according to the output results of the classifier.

5. The method according to claim 4, characterized in that The method of obtaining the predicted lyrics phonemes of each song frame according to the output result of the classifier includes: Obtaining the phoneme probability distribution of each song frame output by the classifier; The predicted lyrics phonemes of each song frame are determined according to the phoneme probability distribution of each song frame.

6. The method according to claim 1, characterized in that Determining the model loss according to the predicted lyrics phonemes and the phoneme labels includes: According to the phoneme labels of the lyrics in the song sample, obtaining the phoneme labels of the lyrics of each song frame; The model loss is determined based on the predicted lyric phonemes for each song frame and the phoneme labels of the lyrics for each song frame.

7. A lyrics recognition method, characterized in that: The method comprises: Acquire the song spectrum information of the song whose lyrics are to be identified and the accompaniment spectrum information of the song; Inputting the song spectrum information and the accompaniment spectrum information of the song into the lyrics acoustic model to obtain the predicted lyrics phonemes of the song output by the lyrics acoustic model; the lyrics acoustic model is trained according to the method according to any one of claims 1 to 6; The predicted lyrics phonemes of the song are input into a language model to obtain the lyrics of the song output by the language model.

8. The method according to claim 7, characterized in that The method further comprises: Acquire multiple lyrics texts from a preset song database, and acquire lyrics phonemes of each lyrics text character in the lyrics texts; Based on the multiple lyrics texts, determining the associated character of each lyrics text character and the occurrence probability of the associated character; acquiring the lyrics phoneme of the associated character from the lyrics phoneme of each lyrics text character in the lyrics text; The language model is constructed based on the lyrics phonemes of each lyrics text character, the lyrics phonemes of the associated characters of each lyrics text character and the occurrence probability of the associated characters.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 or 7 to 8 are implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 or 7 to 8 are implemented.

Images