Method for training rhyme lyrics model, method and device for generating rhyme lyrics

Abstract

The embodiment of the application discloses a training method of a rhyming lyrics model, a generation method and device of rhyming lyrics, and is used in the technical field of lyrics processing. The method comprises the following steps: determining an initial phoneme mapping relationship between a first pronunciation dictionary and a second pronunciation dictionary corresponding to a song sample based on a first phoneme sequence and a second phoneme sequence; checking the initial phoneme mapping relationships corresponding to a plurality of song samples with each other to obtain a target phoneme mapping relationship, and taking the target phoneme mapping relationship as a rhyming lyrics model. By determining the phoneme mapping relationship between the pronunciation dictionaries and taking the phoneme mapping relationship between the pronunciation dictionaries as the rhyming lyrics model, the rhyming lyrics model can be used to convert the song lyrics corresponding to the first pronunciation dictionary into the rhyming lyrics corresponding to the second pronunciation dictionary, so that the song lyrics corresponding to the first pronunciation dictionary can be more comprehensively converted into the rhyming lyrics corresponding to the second pronunciation dictionary.

Description

TECHNICAL FIELD

[0001] Embodiments of the present application relate to the technical field of lyrics processing, in particular to a training method of a homophonic lyrics model, a generation method and device of homophonic lyrics. BACKGROUND

[0002] Existing a cappella lyrics is a kind of homophonic translation technology, which can be used to enable people who do not understand other languages to learn the pronunciation of other languages through Chinese pronunciation. A cappella is a form of music cover, which refers to the conversion of lyrics or melody in a song into another language or melody, making it sound like another song. It is common in guessing games, such as "I love you" in Korean, which is "I love you" in Chinese.

[0003] The existing generation method of a cappella lyrics is a singing ASR technology, which converts the lyrics of a preset language into homophonic lyrics of another language by the similarity of pronunciation between the two languages. However, there may be a low similarity between the pronunciations of the two languages, and when matching by the similarity of pronunciation, if a series of pronunciations cannot find an approximate matching result, it will be considered as low confidence noise and cannot be converted. It can be seen that the existing generation method of a cappella lyrics generates homophonic lyrics by the similarity of pronunciation, which is not comprehensive enough. SUMMARY

[0004] Embodiments of the present application provide a training method of a homophonic lyrics model, a generation method and device of homophonic lyrics, which can more comprehensively convert song lyrics into corresponding homophonic lyrics.

[0005] The present application provides a training method of a homophonic lyrics model, comprising:

[0006] Obtaining lyrics texts corresponding to a plurality of song samples, and extracting a target dry audio of each of the song samples;

[0007] For each of the song samples, converting the lyrics text into a first phoneme state sequence based on a first pronunciation dictionary, and converting the lyrics text into a second phoneme state sequence based on a second pronunciation dictionary; wherein the phoneme state sequence includes a plurality of phoneme states corresponding to audio frames of the target dry audio, and the first pronunciation dictionary and the second pronunciation dictionary use different phoneme conversion rules;

[0008] Determine the first state transition probability between adjacent two frames of phoneme states in the first phoneme state sequence, and the second state transition probability between adjacent two frames of phoneme states in the second phoneme state sequence;

[0009] ​determine a first assignment probability of each frame of audio in the target dry audio being assigned to each phoneme state in the first phoneme state sequence, and a second assignment probability of each frame of audio in the target dry audio being assigned to each phoneme state in the second phoneme state sequence;

[0010] obtain a first phoneme sequence corresponding to the lyrics text based on the first state transition probability and the first assignment probability, and obtain a second phoneme sequence corresponding to the lyrics text based on the second state transition probability and the second assignment probability;

[0011] determine an initial phoneme mapping relationship between the first pronunciation dictionary and the second pronunciation dictionary corresponding to the song sample based on the first phoneme sequence and the second phoneme sequence;

[0012] cross-check the initial phoneme mapping relationships corresponding to the plurality of song samples with each other to obtain a target phoneme mapping relationship, and take the target phoneme mapping relationship as a rhyming lyrics model.

[0013] Further, the converting the lyrics text into the first phoneme state sequence based on the first pronunciation dictionary comprises:

[0014] converting the lyrics text into a phoneme sequence based on the first pronunciation dictionary;

[0015] converting the phoneme sequence into the first phoneme state sequence based on the phoneme state corresponding to each phoneme in the phoneme sequence.

[0016] Further, the determining the first state transition probability between adjacent two frames of phoneme states in the first phoneme state sequence comprises:

[0017] inputting the first phoneme state sequence into a pre-set language model, taking a plurality of phoneme states of adjacent preset frames as a multi-phoneme state group in the language model, determining a total number of phoneme states in the first phoneme state sequence and a number of occurrences of the multi-phoneme state group in the first phoneme state sequence;

[0018] dividing the number of occurrences by the total number of phoneme states to obtain an occurrence probability of the multi-phoneme state group in the first phoneme state sequence;

[0019] taking the occurrence probability as the first state transition probability between adjacent two frames of phoneme states in the multi-phoneme state group.

[0020] Further, the determining the first assignment probability of each frame of audio in the target dry audio being assigned to each phoneme state in the first phoneme state sequence comprises:

[0021] obtaining a preset song corresponding to the first pronunciation dictionary, a preset lyric text corresponding to the preset song, and a preset dry audio of the preset song;

[0022] training an acoustic model based on an audio feature of the preset dry audio and a phoneme state sequence corresponding to the preset lyric text;

[0023] extracting an audio feature of the target dry audio, inputting the audio feature of the target dry audio and a phoneme state in the first phoneme state sequence into the trained acoustic model to obtain the first assignment probability.

[0024] Further, the obtaining the first phoneme sequence corresponding to the lyric text based on the first state transition probability and the first assignment probability comprises:

[0025] determining a maximum probability of phoneme state transition in a plurality of frames of audio corresponding to the target dry audio and a state transition path corresponding to the maximum probability based on the first state transition probability and the first assignment probability;

[0026] extracting a phoneme state corresponding to each frame of audio in the state transition path to obtain a target phoneme state sequence;

[0027] taking the target phoneme state sequence as the first phoneme sequence corresponding to the lyric text.

[0028] Further, the determining the initial phoneme mapping relationship between the first pronunciation dictionary and the second pronunciation dictionary corresponding to the song sample based on the first phoneme sequence and the second phoneme sequence comprises:

[0029] determining a matching number between a first phoneme in the first phoneme sequence and a second phoneme in the second phoneme sequence;

[0030] if the matching number satisfies a preset mapping condition, determining that the first phoneme and the second phoneme have a mapping relationship, and further determining the initial phoneme mapping relationship between the first pronunciation dictionary and the second pronunciation dictionary corresponding to the song sample.

[0031] Further, the mutual checking of the initial phoneme mapping relationships corresponding to the plurality of song samples to obtain a target phoneme mapping relationship comprises:

[0032] determining a mapping number of the same mapping relationship from the initial phoneme mapping relationships corresponding to the plurality of song samples;

[0033] determining a mapping weight of each initial phoneme mapping relationship based on the mapping number, and obtaining the target phoneme mapping relationship based on the mapping weight.

[0034] The embodiment of the present application also provides a method for generating a harmony lyric, comprising:

[0035] obtaining a lyric text corresponding to a target song and the harmony lyric model, wherein the harmony lyric model comprises a phoneme mapping relationship between a first pronunciation dictionary and a second pronunciation dictionary;

[0036] converting the lyric text into a first phoneme sequence based on the first pronunciation dictionary;

[0037] inputting the first phoneme sequence into the harmony lyric model to obtain a second phoneme sequence output by the harmony lyric model;

[0038] converting the second phoneme sequence into a harmony lyric corresponding to the second pronunciation dictionary.

[0039] The embodiment of the present application also provides a training device for a harmony lyric model, comprising:

[0040] a central processing unit, a memory and an input-output interface;

[0041] the memory is a short-term storage memory or a persistent storage memory;

[0042] the central processing unit is configured to communicate with the memory and execute instruction operations in the memory to perform the method.

[0043] The embodiment of the present application also provides a computer readable storage medium comprising instructions, when the instructions are run on a computer, the computer executes the method as described above.

[0044] From the above technical solutions, the embodiment of the present application has the following advantages:

[0045] In the embodiments of the present application, for each song sample, the lyrics text is converted into a first phoneme state sequence based on a first pronunciation dictionary and converted into a second phoneme state sequence based on a second pronunciation dictionary; a first phoneme sequence corresponding to the lyrics text is obtained based on a first state transition probability of the first phoneme state sequence and a first allocation probability; a second phoneme sequence corresponding to the lyrics text is obtained based on a second state transition probability of the second phoneme state sequence and a second allocation probability; an initial phoneme mapping relationship between the first pronunciation dictionary and the second pronunciation dictionary corresponding to the song sample is determined based on the first phoneme sequence and the second phoneme sequence; the initial phoneme mapping relationships corresponding to the plurality of song samples are cross-checked to obtain a target phoneme mapping relationship, and the target phoneme mapping relationship is taken as a euphonic lyrics model. By determining the phoneme mapping relationship between the pronunciation dictionaries and taking the phoneme mapping relationship between the pronunciation dictionaries as the euphonic lyrics model, the euphonic lyrics model can be used to convert the song lyrics corresponding to the first pronunciation dictionary into euphonic lyrics corresponding to the second pronunciation dictionary, so that the song lyrics corresponding to the first pronunciation dictionary can be more comprehensively converted into euphonic lyrics corresponding to the second pronunciation dictionary. BRIEF DESCRIPTION OF DRAWINGS

[0046] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed in the embodiment description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments described in the present application, and other drawings can also be obtained by those skilled in the art based on these drawings.

[0047] Figure 1 A communication network architecture diagram disclosed by the embodiments of the present application;

[0048] Figure 2 A euphonic lyrics model training flowchart disclosed by the embodiments of the present application;

[0049] Figure 3 Another euphonic lyrics model training flowchart disclosed by the embodiments of the present application;

[0050] Figure 4 A euphonic lyrics generation diagram disclosed by the embodiments of the present application;

[0051] Figure 5 A euphonic lyrics model training device diagram disclosed by the embodiments of the present application;

[0052] Figure 6 Another euphonic lyrics model training device diagram disclosed by the embodiments of the present application. DETAILED DESCRIPTION

[0053] In order to make the purposes, technical solutions and advantages of the present application clearer, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative work fall within the scope of protection of the present application.

[0054] In the description of the embodiments of the present application, it should be noted that the terms "center", "upper", "lower", "left", "right", "vertical", "horizontal", "inner", "outer" and the like indicate the orientation or positional relationship shown in the drawings, and are only for the convenience of describing the embodiments of the present application and simplifying the description, and do not indicate or imply that the device or element referred to must have a particular orientation, be constructed and operated in a particular orientation, and therefore cannot be understood as a limitation on the embodiments of the present application.

[0055] In the description of the embodiments of the present application, it should be noted that unless otherwise explicitly specified and limited, the terms "mounting", "connecting", "connection" should be understood broadly, for example, can be fixedly connected, or can be detachably connected, or integrally connected; can be mechanically connected, or can be electrically connected; can be directly connected, or can be indirectly connected through an intermediate medium; can be the communication inside two elements. For those of ordinary skill in the art, the specific meanings of the above terms in the embodiments of the present application can be understood according to the specific circumstances.

[0056] The existing harmonic lyrics generation method is as follows Figure 1As shown, the homophonic lyrics generation device 101 can acquire song lyrics from the audio playback device 102 and then convert the song lyrics into homophonic lyrics. The song lyrics and the homophonic lyrics are in different languages. Homophony generally utilizes the condition of similar pronunciation, using homophones or near-homophones to replace the original characters, creating a rhetorical expression with a catchy tone. These homophonic lyrics can be understood as lyrics that are merely for euphony. For example, if the song lyrics are in Korean, the homophonic lyrics generation device 101 can convert Korean lyrics into Chinese homophonic lyrics, or German song lyrics into English homophonic lyrics. It is understood that the homophonic lyrics generation device 101 can be connected to one or more audio players 102 to convert the song lyrics in the audio player 102 into corresponding homophonic lyrics. Existing homophonic lyrics generation methods convert lyrics in a preset language into homophonic lyrics in another language based on the similarity of pronunciation between the two languages. However, there may be cases where the pronunciation between the two languages ​​has a low similarity. When matching based on pronunciation similarity, a series of pronunciations that cannot find an approximate matching result will be considered low-confidence noise and cannot be converted. It is evident that existing methods for generating homophonic lyrics based on pronunciation similarity are insufficient. Therefore, this application provides a method for training a homophonic lyrics model, which can more comprehensively convert song lyrics into corresponding homophonic lyrics, such as... Figure 2 As shown, the specific steps include the following:

[0057] 201. Obtain the lyrics text corresponding to multiple song samples, and extract the target dry audio of each song sample.

[0058] In this embodiment, the homophonic lyrics generation device can generate lyrics text corresponding to multiple song samples and extract the target dry audio of each song sample. The multiple song samples include songs in one or more languages, which is not specifically limited here. The song samples can be Chinese songs, English songs, or Korean songs, and correspondingly, the lyrics text of the song samples can be Chinese text, English text, or Korean text, which is not specifically limited here. Specifically, the song audio corresponding to the song sample can be obtained, and dry audio extraction can be performed on the song audio, that is, extracting the human voice from the song audio to obtain the target dry audio. For example, the dry audio in the song audio can be extracted using the Spleeter model. The target dry audio includes the human voice audio corresponding to the target song, but does not include the accompaniment audio. The duration of the extracted target dry audio is the same as the duration of the song audio, and the lyrics text is the text corresponding to the song sample.

[0059] Optionally, since the lyrics text may include some non-lyrics information such as lyrics writing and music writing irrelevant to singing, the lyrics text can be obtained after performing non-lyrics information filtering processing on the initial lyrics text of the song sample. That is, obtaining the initial lyrics text corresponding to the song sample; performing non-lyrics information filtering processing on the initial lyrics text to obtain the lyrics text. The non-lyrics information generally appears in the beginning part of the lyrics text, and the non-lyrics information filtering processing can set corresponding filtering rules according to the information characteristics of the non-lyrics information, for example, by detecting keywords, key symbols, etc., to perform non-lyrics information filtering processing on the initial lyrics text, delete the non-lyrics information in the initial lyrics text, and only keep the text corresponding to the lyrics of the song sample, that is, keep the lyrics text relevant to singing.

[0060] 202. convert the lyrics text into a first phoneme state sequence based on the first pronunciation dictionary and convert the lyrics text into a second phoneme state sequence based on the second pronunciation dictionary.

[0061] After obtaining the lyrics text of the song sample, for each song sample, the lyrics text can be converted into a first phoneme state sequence based on the first pronunciation dictionary and converted into a second phoneme state sequence based on the second pronunciation dictionary. Among them, the phoneme state sequence includes a plurality of phoneme states corresponding to the audio frames of the target dry audio, that is, each phoneme state corresponds to a frame of audio in the target dry audio. It can be understood that the phoneme is the smallest unit of speech divided according to the natural properties of language, and one pronunciation action forms one phoneme; for example, the pronunciation of the three Chinese characters "I and you" can be represented by "w, o, h, e, n, i" respectively. The phonemes corresponding to the pronunciation of English can be represented by phonetic symbols such as / i / , / I / , / e / , / / , etc. The pronunciation dictionary can record the mapping relationship between characters (words) and phonemes, that is, the phonemes corresponding to each character (word). The first pronunciation dictionary and the second pronunciation dictionary can be Chinese pronunciation dictionary, English pronunciation dictionary, etc., and the first pronunciation dictionary and the second pronunciation dictionary are pronunciation dictionaries of different languages, and the pronunciation dictionaries of different languages adopt different phoneme conversion rules, that is, the same lyrics text converted by different pronunciation dictionaries has different phonemes, for example, the phonemes converted by the Chinese pronunciation dictionary are generally represented by pinyin, and the phonemes converted by the English pronunciation dictionary are generally represented by phonetic symbols.

[0062] The phoneme state (HMM state) is a more detailed speech unit obtained by dividing each phoneme. Generally, one phoneme can correspond to three phoneme states, that is, the initial sound, the sustained sound and the ending sound of the pronunciation of the phoneme can be determined as the three phoneme states corresponding to the phoneme. Based on the mapping relationship between the characters and the phonemes in the first pronunciation dictionary, the phonemes corresponding to each character in the lyrics text can be determined, and the phoneme set corresponding to the lyrics text can be obtained. Then, the phonemes in the phoneme set can be sorted according to the character order of each character in the lyrics text or according to the pronunciation order of the phoneme pronunciation time in the target dry sound frequency, to obtain the phoneme sequence corresponding to the lyrics text, and then the phonemes in the phoneme sequence are converted into states to obtain the corresponding first phoneme state sequence.

[0063] When the lyrics text is converted into the second phoneme state sequence based on the second pronunciation dictionary, the lyrics text is converted into the corresponding phoneme sequence according to the corresponding relationship between the characters and the phonemes in the second pronunciation dictionary, and then the phoneme sequence is converted into the second phoneme state sequence. It can be understood that the first pronunciation dictionary and the second pronunciation dictionary adopt different phoneme conversion rules, that is, the corresponding relationship between the characters and the phonemes in the first pronunciation dictionary and the second pronunciation dictionary is different, that is, the phoneme sequence obtained by converting the lyrics text through the first pronunciation dictionary is different from the phoneme sequence obtained by converting the lyrics text through the second pronunciation dictionary, and the first phoneme state sequence and the second phoneme state sequence obtained correspondingly also have differences.

[0064] 203, determine the first state transition probability of the first phoneme state sequence and the second state transition probability of the second phoneme state sequence.

[0065] After obtaining the first phoneme state sequence and the second phoneme state sequence, the first state transition probability of the first phoneme state sequence and the second state transition probability of the second phoneme state sequence can be determined, that is, the first state transition probability between the adjacent two frames of phoneme states in the first phoneme state sequence and the second state transition probability between the adjacent two frames of phoneme states in the second phoneme state sequence. It can be understood that the phoneme state sequence includes a plurality of phoneme states, and one phoneme state corresponds to one frame of audio, and a plurality of phoneme states are distributed in order on one frame of audio. The state transition probability refers to the probability that the phoneme state corresponding to the current frame in the phoneme state sequence is transferred to the phoneme state corresponding to the next frame. It can be understood that the transition of the phoneme state can be the transition of two different phoneme states, that is, the phoneme states corresponding to the current frame and the next frame are different phoneme states, or it can be the transition of the same phoneme state, that is, the phoneme states corresponding to the current frame and the next frame are the same phoneme state, which is not limited here.

[0066] 204、determine a first assignment probability of each frame of audio in the target dry audio being assigned to the first sequence of phonemes, and a second assignment probability of each frame of audio in the target dry audio being assigned to the second sequence of phonemes.

[0067] In the embodiments of the present application, the first assignment probability of each frame of audio in the target dry audio being assigned to each phoneme state in the first sequence of phonemes, and the second assignment probability of each frame of audio in the target dry audio being assigned to each phoneme state in the second sequence of phonemes can also be determined. The assignment probability refers to the probability of being assigned to a phoneme state in the sequence of phonemes in each frame of audio in the target dry audio. Specifically, the target dry audio and the first sequence of phonemes converted based on the first pronunciation dictionary can be input into a pre-trained acoustic model, and the acoustic model outputs the first assignment probability of each frame of audio in the target dry audio being assigned to each phoneme state in the first sequence of phonemes; the target dry audio and the second sequence of phonemes converted based on the second pronunciation dictionary can be input into the pre-trained acoustic model, and the acoustic model outputs the second assignment probability of each frame of audio in the target dry audio being assigned to each phoneme state in the second sequence of phonemes.

[0068] 205、based on the first state transition probability and the first assignment probability, obtain the first phoneme sequence corresponding to the lyrics text; based on the second state transition probability and the second assignment probability, obtain the second phoneme sequence corresponding to the lyrics text.

[0069] In the embodiments of the present application, the first phoneme sequence corresponding to the lyrics text can be obtained based on the first state transition probability and the first assignment probability; the second phoneme sequence corresponding to the lyrics text can be obtained based on the second state transition probability and the second assignment probability. Specifically, the first state transition probability is the transition probability between adjacent phonemes in the first sequence of phonemes, and the first assignment probability is the assignment probability of each frame of audio in the target dry audio being assigned to each phoneme state in the first sequence of phonemes. By the first state transition probability and the first assignment probability, the phoneme state corresponding to each frame of audio in the target dry audio can be determined, the corresponding sequence of phoneme states is obtained, and the phonemes in the sequence of phoneme states are inversely converted into phonemes to obtain the first phoneme sequence corresponding to the lyrics text. Based on the second state transition probability and the second assignment probability, the second phoneme sequence corresponding to the lyrics text is obtained, that is, by the second state transition probability and the second assignment probability, the phoneme state corresponding to each frame of audio in the target dry audio can be determined, the corresponding sequence of phoneme states is obtained, and the phonemes in the sequence of phoneme states are inversely converted into phonemes to obtain the second phoneme sequence corresponding to the lyrics text.

[0070] 206、based on the first phoneme sequence and the second phoneme sequence, determine the phoneme mapping relationship between the first pronunciation dictionary and the second pronunciation dictionary corresponding to the song sample.

[0071] After obtaining the first phoneme sequence and the second phoneme sequence corresponding to the lyrics text, the phoneme mapping relationship between the first pronunciation dictionary and the second pronunciation dictionary corresponding to the song sample can be determined based on the first phoneme sequence and the second phoneme sequence. It can be understood that the first pronunciation dictionary records the pronunciation phonemes of the corresponding language, and the second pronunciation dictionary records the pronunciation phonemes of another language; the first phoneme sequence is the phoneme sequence obtained by converting the lyrics text of the song sample based on the first pronunciation dictionary, and the second phoneme sequence is the phoneme sequence obtained by converting the lyrics text of the same song sample based on the second pronunciation dictionary; the phonemes in the first phoneme sequence and the phonemes in the second phoneme sequence can be one-to-one corresponding based on the phoneme pronunciation order, that is, the phoneme mapping relationship between the phonemes of the song sample in the first pronunciation dictionary and the phonemes of the song sample in the second pronunciation dictionary can be obtained. The phoneme mapping relationship can be understood as the phonemes of the first pronunciation dictionary corresponding to the phonemes of the second pronunciation dictionary.

[0072] 207、The initial phoneme mapping relationships corresponding to the plurality of song samples are verified with each other to obtain a euphonic lyrics model.

[0073] After determining the initial phoneme mapping relationship between the first pronunciation dictionary and the second pronunciation dictionary corresponding to the song sample, the initial phoneme mapping relationships corresponding to the plurality of song samples can be verified with each other to obtain a euphonic lyrics model. That is, by comparing a plurality of initial phoneme mapping relationships, a more accurate phoneme mapping relationship can be obtained, and a corresponding phoneme mapping dictionary, i.e., a euphonic lyrics model, can be obtained.

[0074] In an implementable manner, after obtaining the rhyming lyrics model, a lyrics text corresponding to a target song can be obtained, the target song corresponding to the first pronunciation dictionary, i.e., the lyrics text corresponding to the target song is the song lyrics corresponding to the first pronunciation dictionary; the lyrics text is converted into a first phoneme sequence based on the first pronunciation dictionary; the first phoneme sequence is input into the rhyming lyrics model to obtain a second phoneme sequence output by the rhyming lyrics model; and the second phoneme sequence is converted into rhyming lyrics corresponding to a second pronunciation dictionary. That is, the rhyming lyrics model is used to convert the song lyrics corresponding to the first pronunciation dictionary into rhyming lyrics corresponding to the second pronunciation dictionary, so as to realize the generation of the a cappella lyrics. The song lyrics corresponding to the first pronunciation dictionary are lyrics in the same language as the first pronunciation dictionary, such as Korean lyrics corresponding to a Korean pronunciation dictionary, and the rhyming lyrics corresponding to the second pronunciation dictionary are lyrics in the same language as the second pronunciation dictionary, such as Chinese lyrics corresponding to a Chinese pronunciation dictionary. The song lyrics can be converted into a corresponding phoneme sequence based on the first pronunciation dictionary, and the phoneme sequence can be mapped into a phoneme sequence corresponding to the second pronunciation dictionary based on the phoneme mapping relationship in the rhyming lyrics model, so as to obtain the rhyming lyrics corresponding to the second pronunciation dictionary. For example, the Korean lyrics can be converted into a corresponding phoneme sequence based on the Korean pronunciation dictionary, and the phoneme sequence can be mapped into a phoneme sequence corresponding to the Chinese pronunciation dictionary based on the phoneme mapping relationship between the Korean pronunciation dictionary and the Chinese pronunciation dictionary, so as to obtain corresponding Chinese rhyming lyrics.

[0075] It can be seen that, in the embodiments of the present application, for each song sample, the lyrics text is converted into a first phoneme state sequence based on the first pronunciation dictionary and is converted into a second phoneme state sequence based on the second pronunciation dictionary; the first phoneme state sequence is converted into a first phoneme sequence based on the first state transition probability and the first allocation probability of the first phoneme state sequence; the second phoneme state sequence is converted into a second phoneme sequence based on the second state transition probability and the second allocation probability of the second phoneme state sequence; the initial phoneme mapping relationship between the first pronunciation dictionary and the second pronunciation dictionary corresponding to the song sample is determined based on the first phoneme sequence and the second phoneme sequence; and the initial phoneme mapping relationships corresponding to the plurality of song samples are cross-checked to obtain a target phoneme mapping relationship, which is taken as the rhyming lyrics model. By determining the phoneme mapping relationship between the pronunciation dictionaries and taking the phoneme mapping relationship between the pronunciation dictionaries as the rhyming lyrics model, the rhyming lyrics model can be used to convert the song lyrics corresponding to the first pronunciation dictionary into rhyming lyrics corresponding to the second pronunciation dictionary, and the song lyrics corresponding to the first pronunciation dictionary can be more comprehensively converted into rhyming lyrics corresponding to the second pronunciation dictionary.

[0076] Further, in the embodiments of the present application, the song ASR model can be used to realize the generation of the rhyming lyrics, as shown in the following formula (1): Figure 3 Specifically, the method comprises the following steps:

[0077] 301、obtain lyrics texts corresponding to a plurality of song samples, and extract target dry audio of each song sample.

[0078] 302、convert the lyrics texts into a first phoneme state sequence based on a first pronunciation dictionary, and convert the lyrics texts into a second phoneme state sequence based on a second pronunciation dictionary.

[0079] It can be understood that steps 301 and 302 are similar to steps 201 and 202 described above, and details are not repeated here.

[0080] 303、obtain a first state transition probability of the first phoneme state sequence and a second state transition probability of the second phoneme state sequence based on a pre-set language model.

[0081] In the embodiments of the present application, the first state transition probability of the first phoneme state sequence and the second state transition probability of the second phoneme state sequence can be obtained based on a pre-set language model. The pre-set language model is a statistical model, such as an n-gram model. The pre-set language model can be used to statistically obtain the first state transition probability between adjacent two frames of phoneme states in the first phoneme state sequence and the second state transition probability between adjacent two frames of phoneme states in the second phoneme state sequence.

[0082] Specifically, the phoneme state sequence can be input into the pre-set language model, and in the language model, a plurality of phoneme states of adjacent preset frames are taken as a plurality of triphone state groups, and the total number of phoneme states in the phoneme state sequence and the number of occurrences of the triphone state groups in the phoneme state sequence are determined; for example, a 3-gram model can be used, one letter represents one phoneme state, and adjacent “ABC” in the phoneme state sequence is taken as a triphone state group, the number of occurrences of “ABC” in the phoneme state sequence and the total number of phoneme states in the phoneme state sequence can be determined. The number of occurrences is divided by the total number of phoneme states, and the occurrence probability of the triphone state group in the phoneme state sequence is obtained, which is taken as the state transition probability between adjacent two frames of phoneme states in the triphone state group. Through the pre-set language model, the first state transition probability of the first phoneme state sequence and the second state transition probability of the second phoneme state sequence can be obtained.

[0083] 304、obtain a first distribution probability and a second distribution probability of each frame of audio in the target dry audio based on a pre-trained acoustic model.

[0084] In the embodiments of the present application, the first distribution probability and the second distribution probability of each frame of audio in the target dry audio can be obtained based on a pre-trained acoustic model. The first distribution probability is the belonging probability of each frame of audio to each phoneme state in the first phoneme state sequence, and the second distribution probability is the belonging probability of each frame of audio to each phoneme state in the second phoneme state sequence. The acoustic model can be a GMM acoustic model or a DNN acoustic model, which is not limited here. The audio features of the target dry audio and the phoneme states in the first phoneme state sequence are input into the trained acoustic model to obtain the first distribution probability, and the audio features of the target dry audio and the phoneme states in the second phoneme state sequence are input into the trained acoustic model to obtain the second distribution probability. It can be understood that the acoustic model input by the first phoneme state sequence and the acoustic model input by the second phoneme state sequence are different acoustic models.

[0085] Specifically, the first distribution probability of each frame of audio in the target dry audio to each phoneme state in the first phoneme state sequence can be obtained by obtaining a first song corresponding to the first pronunciation dictionary, a first song text corresponding to the first song, and a first dry audio of the first song. The first song is a song in the same language as the first pronunciation dictionary. The acoustic model is trained based on the audio features of the first dry audio and the phoneme state sequence corresponding to the first song text. That is, the audio features of the first dry audio and the phoneme state sequence corresponding to the first song text are used as training samples, and the acoustic model is monophone trained based on the training samples to obtain a first acoustic model. The first acoustic model is triphone trained based on the training samples to obtain a trained acoustic model. It can be understood that the audio features can be mel-frequency cepstral coefficient (MFCC) features, mel-spectrum features, or HPCP features, which are not limited here. The audio features of the target dry audio are extracted, and the audio features of the target dry audio and the phoneme states in the first phoneme state sequence are input into the trained acoustic model to obtain the first distribution probability.

[0086] For the acoustic model input by the second phoneme state sequence, a second song corresponding to the second pronunciation dictionary, a second song text corresponding to the second song, and a second dry audio of the second song are obtained, and the audio features of the second dry audio and the phoneme state sequence corresponding to the second song text are used as training samples of the acoustic model. It can be understood that the language model and the trained acoustic model can be understood as a song ASR model, i.e., a DNN-HMM song ASR system. For example, for a Korean pronunciation dictionary, a Korean song ASR model can be obtained, and for a Chinese pronunciation dictionary, a Chinese song ASR model can be obtained.

[0087] 305、determine a state transition path corresponding to the maximum probability based on the state transition probability and the allocation probability, to obtain the first phoneme sequence and the second phoneme sequence corresponding to the lyrics text.

[0088] In the embodiments of the present application, the state transition path corresponding to the maximum probability can be determined based on the state transition probability and the allocation probability, to obtain the first phoneme sequence and the second phoneme sequence corresponding to the lyrics text.

[0089] Specifically, the maximum probability of phoneme state transition in the multi-frame audio corresponding to the target dry audio and the state transition path corresponding to the maximum probability can be determined according to the first state transition probability and the first allocation probability. For example, the sum of the product of the allocation probability of the phoneme state of the current frame and the state transition probability of the phoneme state transition from the phoneme state of the current frame to the phoneme state of the next frame can be calculated to obtain the maximum probability of phoneme state transition. That is, the maximum probability path (state transition path) can be found by forced alignment decoding in the state transition corresponding to the current lyrics text. The phoneme state corresponding to each frame of audio in the state transition path is extracted to obtain a target phoneme state sequence. The target phoneme state sequence is taken as the first phoneme sequence corresponding to the lyrics text.

[0090] It can be understood that the maximum probability of phoneme state transition in the multi-frame audio corresponding to the target dry audio and the state transition path corresponding to the maximum probability can be determined according to the second state transition probability and the second allocation probability. The corresponding phoneme state sequence is obtained based on the state transition path, and the phoneme state sequence is converted into the second phoneme sequence corresponding to the lyrics text. That is, the first phoneme sequence corresponding to the first pronunciation dictionary and the second phoneme sequence corresponding to the second pronunciation dictionary can be output by the singing ASR model.

[0091] 306、determine the initial phoneme mapping relationship between the first pronunciation dictionary and the second pronunciation dictionary based on the matching number of phonemes between the first phoneme sequence and the second phoneme sequence.

[0092] After obtaining the first phoneme sequence and the second phoneme sequence, the phoneme mapping relationship between the first pronunciation dictionary and the second pronunciation dictionary can be determined based on the matching number of phonemes between the first phoneme sequence and the second phoneme sequence. Specifically, the matching number between the first phoneme in the first phoneme sequence and the second phoneme in the second phoneme sequence is determined. If the matching number meets a preset mapping condition, it is determined that the first phoneme and the second phoneme have a mapping relationship, and then the initial phoneme mapping relationship between the first pronunciation dictionary and the second pronunciation dictionary is determined. It can be understood that the matching number meeting the preset mapping condition means that the matching number between the first phoneme and the second phoneme is greater than the matching number between the first phoneme and other phonemes in the second phoneme sequence.

[0093] For example, Figure 4As shown, when the first pronunciation dictionary is a Korean pronunciation dictionary, a Korean acoustic model can be trained based on the audio features (MFCC features) of the dry audio of a Korean song and the phoneme state sequence (HMM state sequence) of the corresponding Korean lyrics as training samples, and a Korean song ASR model (DNN-HMM Korean song ASR system) can be obtained based on the Korean acoustic model and the language model. When the second pronunciation dictionary is a Chinese pronunciation dictionary, a Chinese acoustic model can be trained based on the audio features (MFCC features) of the dry audio of a Chinese song and the phoneme state sequence (HMM state sequence) of the corresponding Chinese lyrics as training samples, and a Chinese song ASR model (DNN-HMM Chinese song ASR system) can be obtained based on the Chinese acoustic model and the language model. The audio features of the dry audio corresponding to the Korean song and the phoneme state sequence corresponding to the lyrics of the Korean song are input into the Korean song ASR model and the Chinese song ASR model, respectively, and based on the output phoneme state sequence, a Korean phoneme sequence and a Chinese phoneme sequence can be obtained. By matching the Korean phoneme sequence with the Chinese phoneme sequence, a phoneme mapping relationship between the Korean pronunciation dictionary and the Chinese pronunciation dictionary (i.e., constructing a mapping dictionary) can be obtained. For example, if the Korean phoneme sequence is co1 / co3 / co1 / co4 / co2 / co1 and the Chinese phoneme sequence is ch1 / ch3 / ch4 / ch6 / ch2 / ch1, based on the matching statistical mapping relationship between the phonemes, the following table can be obtained:

[0094] col-chi 2 times co3-ch3 1 time col-ch4 1 time co4-ch6 1 time co2-ch2 1 time

[0095] As can be seen from the table, the matching times of the Korean phoneme co1 and the Chinese phoneme ch1 is 2, and the matching times of the Korean phoneme co1 and the Chinese phoneme ch4 is 1. The maximum matching times are determined, i.e., the phoneme mapping relationship between the Korean phoneme co1 and the Chinese phoneme ch1 is determined, and the initial phoneme mapping relationship (mapping dictionary) between the Korean pronunciation dictionary and the Chinese pronunciation dictionary is obtained.

[0096] 307、Based on the mapping number of the same mapping relationship in the initial phoneme mapping relationship, a harmony song model is obtained.

[0097] It can be understood that, in order to improve the accuracy of the phoneme mapping relationship between the first pronunciation dictionary and the second pronunciation dictionary, a large number of song samples are needed. The lyrics text of the song samples is matched based on the first phoneme sequence obtained based on the first pronunciation dictionary and the second phoneme sequence obtained based on the second pronunciation dictionary, a plurality of initial phoneme mapping relationships between the first pronunciation dictionary and the second pronunciation dictionary are obtained, and based on the mapping number of the same mapping relationship in the initial phoneme mapping relationship, a harmony song model is obtained.

[0098] Specifically, the mapping quantity of the same mapping relationship can be determined from the initial phoneme mapping relationships corresponding to the plurality of song samples; the mapping weight of each initial phoneme mapping relationship is determined based on the mapping quantity, and the target phoneme mapping relationship is obtained based on the mapping weight. That is, for each phoneme in the pronunciation dictionary, the initial phoneme mapping relationship with the maximum mapping weight is found as the target phoneme mapping relationship corresponding to the phoneme, and the corresponding harmony lyrics model is obtained. After obtaining the harmony lyrics model, the corresponding Chinese phoneme sequence can be obtained by 100% conversion according to the Korean lyrics, the phoneme sequence is converted into Chinese, and the Chinese karaoke lyrics are obtained.

[0099] It can be seen that the present application proposes a karaoke lyrics generation method based on a song ASR model. Taking Korean karaoke lyrics as an example, first, Korean song samples and corresponding pronunciation dictionaries are collected, and a Korean song ASR model is trained. Then, Chinese song samples and corresponding pronunciation dictionaries are collected, and a Chinese song ASR model is trained. Then, a mapping relationship dictionary is constructed. In order to ensure the stability and reliability of the dictionary, a large amount of corpus needs to be prepared for constructing a statistical dictionary. First, the Korean song is input into the Korean ASR model, and the Korean phoneme sequence is recognized. Then, the Korean song is input into the Chinese song ASR system, and the Chinese phoneme sequence is obtained. The phoneme matching probability is counted, the best phoneme matching result is obtained, and the phoneme mapping dictionary is constructed. Finally, the Korean lyrics can be converted into karaoke lyrics through dictionary mapping. By combining different language acoustic models and constructing a mapping dictionary, it can be ensured that the lyrics can generate karaoke lyrics by 100%, and the usability of karaoke lyrics generation is ensured.

[0100] The present application also provides a training device for harmony lyrics model, as shown in Figure 5 , comprising:

[0101] The acquisition unit 501 is configured to acquire lyrics texts corresponding to a plurality of song samples, and extract a target dry audio of each song sample.

[0102] The conversion unit 502 is configured to convert the lyrics text into a first phoneme state sequence based on a first pronunciation dictionary and convert the lyrics text into a second phoneme state sequence based on a second pronunciation dictionary for each song sample. The first pronunciation dictionary and the second pronunciation dictionary use different phoneme conversion rules.

[0103] The first determination unit 503 is configured to determine a first state transition probability between adjacent two frames of phoneme states in the first phoneme state sequence and a second state transition probability between adjacent two frames of phoneme states in the second phoneme state sequence.

[0104] The second determining unit 504 is configured to determine a first allocation probability of each frame of audio in the target dry audio being allocated to each phoneme state in the first phoneme state sequence, and a second allocation probability of each frame of audio in the target dry audio being allocated to each phoneme state in the second phoneme state sequence.

[0105] The executing unit 505 is configured to obtain a first phoneme sequence corresponding to the lyric text based on the first state transition probability and the first allocation probability, and obtain a second phoneme sequence corresponding to the lyric text based on the second state transition probability and the second allocation probability.

[0106] The mapping unit 506 is configured to determine an initial phoneme mapping relationship corresponding to the song sample between the first pronunciation dictionary and the second pronunciation dictionary based on the first phoneme sequence and the second phoneme sequence.

[0107] The checking unit 507 is configured to check the initial phoneme mapping relationships corresponding to a plurality of song samples with each other to obtain a target phoneme mapping relationship, and take the target phoneme mapping relationship as a euphonic lyric model.

[0108] Embodiments of the present application further provide a training device 600 of a euphonic lyric model, as shown in Figure 6 The training device 600 of the euphonic lyric model comprises:

[0109] a central processing unit 601, a memory 602 and an input and output interface 603.

[0110] The memory 602 is a transitory storage memory or a persistent storage memory; wherein the training device 600 of the euphonic lyric model further comprises a wired or wireless network interface 604 and a power supply 605.

[0111] The central processing unit 601 is configured to communicate with the memory 602 and execute instruction operations in the memory 602 to perform the method as described above.

[0112] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the system, device and unit described above can refer to the corresponding processes in the foregoing method embodiments, which will not be described here.

[0113] In several embodiments provided in the present application, it should be understood that the disclosed system, device and method can be implemented in other manners. For example, the described device embodiments are merely schematic. The division of the units is merely a logical function division. There can be another division manner for the actual implementation, for example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the displayed or discussed mutual couplings or direct couplings or communication connections between the units can be indirect couplings or communication connections through some interfaces, devices or units, and can be electrical, mechanical or in other forms.

[0114] The units described as separated components can or can not be physically separated, and the components displayed as units can or can not be physical units, i.e., can be located in one place, or can be distributed on a plurality of network units. Some or all of the units can be selected according to actual needs to achieve the purposes of the embodiments of the present application.

[0115] In addition, each functional unit in the embodiments of the present application can be integrated in one processing unit, or each unit can exist physically, or two or more units can be integrated in one unit. The integrated unit can be implemented in the form of hardware, or in the form of software functional units.

[0116] When the integrated unit is implemented in the form of software functional units and sold or used as an independent product, it can be stored in a computer readable storage medium. Based on such an understanding, the technical solutions of the present application essentially or partly, or all or part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium, and includes a plurality of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in the embodiments of the present application. The foregoing storage medium includes: U disk, mobile hard disk, read-only memory (ROM, read-only memory), random access memory (RAM, random access memory), magnetic disk or optical disk, and various other media that can store program codes.

Claims

1. A training method for a homophonic lyric model, characterized in that, include: Obtain the lyrics text corresponding to multiple song samples, and extract the target dry audio of each song sample; For each song sample, the lyrics text is converted into a first phoneme state sequence based on a first pronunciation dictionary, and the lyrics text is converted into a second phoneme state sequence based on a second pronunciation dictionary; wherein, the phoneme state sequence includes multiple phoneme states corresponding to audio frames of the target dry audio, and the first pronunciation dictionary and the second pronunciation dictionary adopt different phoneme conversion rules; Determine the first state transition probability between two adjacent frames of phoneme states in the first phoneme state sequence, and the second state transition probability between two adjacent frames of phoneme states in the second phoneme state sequence. Determine a first allocation probability for each frame of the target dry audio to be assigned to each phoneme state in the first phoneme state sequence, and a second allocation probability for each frame of the target dry audio to be assigned to each phoneme state in the second phoneme state sequence. Based on the first state transition probability and the first allocation probability, a first phoneme sequence corresponding to the lyrics text is obtained; based on the second state transition probability and the second allocation probability, a second phoneme sequence corresponding to the lyrics text is obtained. Based on the first phoneme sequence and the second phoneme sequence, the initial phoneme mapping relationship between the first pronunciation dictionary and the second pronunciation dictionary is determined for the song sample; The initial phoneme mapping relationships corresponding to multiple song samples are cross-validated to obtain the target phoneme mapping relationship, which is then used as the homophonic lyrics model.

2. The training method according to claim 1, characterized in that, The step of converting the lyrics text into a first phoneme state sequence based on a first pronunciation dictionary includes: Based on the first pronunciation dictionary, the lyrics text is converted into a phoneme sequence; Based on the phoneme state corresponding to each phoneme in the phoneme sequence, the phoneme sequence is converted into the first phoneme state sequence.

3. The training method according to claim 1, characterized in that, The determination of the first state transition probability between two adjacent frames of phoneme states in the first phoneme state sequence includes: The first phoneme state sequence is input into a pre-set language model. In the language model, multiple phoneme states of adjacent preset frames are taken as a multi-phoneme state group. The total number of phoneme states in the first phoneme state sequence and the number of times the multi-phoneme state group appears in the first phoneme state sequence are determined. Divide the number of occurrences by the total number of phoneme states to obtain the probability of occurrence of the multi-phoneme state group in the first phoneme state sequence; The occurrence probability is used as the first state transition probability between two adjacent frame phoneme states in the multi-phoneme state group.

4. The training method according to claim 1, characterized in that, The determination of the first allocation probability of each frame of audio in the target dry audio to each phoneme state in the first phoneme state sequence includes: Obtain the first song corresponding to the first pronunciation dictionary, the first lyrics text corresponding to the first song, and the first dry audio of the first song; An acoustic model is trained based on the audio features of the first dry audio and the phoneme state sequence corresponding to the first lyrics text. The audio features of the target dry audio are extracted, and the audio features of the target dry audio and the phoneme states in the first phoneme state sequence are input into the trained acoustic model to obtain the first allocation probability.

5. The training method according to claim 1, characterized in that, The step of obtaining the first phoneme sequence corresponding to the lyrics text based on the first state transition probability and the first allocation probability includes: Based on the first state transition probability and the first allocation probability, determine the maximum probability of phoneme state transition in the multi-frame audio corresponding to the target dry audio and the state transition path corresponding to the maximum probability. Extract the phoneme state corresponding to each frame of audio in the state transition path to obtain the target phoneme state sequence; The target phoneme state sequence is used as the first phoneme sequence corresponding to the lyrics text.

6. The training method according to claim 1, characterized in that, The step of determining the initial phoneme mapping relationship between the song sample and the first and second phoneme sequences, based on the first and second phoneme sequences, includes: Determine the number of matches between the first phoneme in the first phoneme sequence and the second phoneme in the second phoneme sequence; If the number of matching times meets the preset mapping conditions, it is determined that there is a mapping relationship between the first phoneme and the second phoneme, and then it is determined that the song sample corresponds to the initial phoneme mapping relationship between the first pronunciation dictionary and the second pronunciation dictionary.

7. The training method according to claim 1, characterized in that, The step of cross-validating the initial phoneme mapping relationships corresponding to multiple song samples to obtain the target phoneme mapping relationship includes: From the initial phoneme mapping relationships corresponding to multiple song samples, determine the number of mappings with the same mapping relationship; The mapping weight of each initial phoneme mapping relationship is determined based on the number of mappings, and the target phoneme mapping relationship is obtained based on the mapping weight.

8. A method for generating homophonic lyrics, characterized in that, include: Obtain the lyrics text corresponding to the target song, and the homophonic lyrics model according to any one of claims 1 to 7, wherein the homophonic lyrics model includes a phoneme mapping relationship between a first pronunciation dictionary and a second pronunciation dictionary; the target song corresponds to the first pronunciation dictionary; The lyrics text is converted into a first phoneme sequence based on the first pronunciation dictionary; The first phoneme sequence is input into the homophonic lyrics model to obtain the second phoneme sequence output by the homophonic lyrics model; The second phoneme sequence is converted into homophonic lyrics corresponding to the second pronunciation dictionary.

9. A training device for a homophonic lyric model, characterized in that, include: Central processing unit, memory, and input / output interfaces; The memory is either a short-term storage memory or a persistent storage memory; The central processing unit is configured to communicate with the memory and execute instructions in the memory to perform the method of any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, Includes instructions that, when executed on a computer, cause the computer to perform the method as described in any one of claims 1 to 7.

Images