Rap music automatic generation method and system

Abstract

The application discloses a rap music automatic generation method, which comprises the following steps: obtaining input text, obtaining the sentence segmentation result of the text according to a sentence segmentation technology, filling words in each phrase of accompaniment music according to the beat detection result of the accompaniment music and the sentence segmentation result, obtaining the rap voice audio of each phrase by using a voice synthesis technology according to the filling word result, splicing each rap voice audio, mixing the rap voice audio with the accompaniment music, and outputting the final rap result. The application also discloses a rap music automatic generation system, an electronic device and a computer readable storage medium. Therefore, the rap music that completely fits can be automatically generated.

Description

Technical Field

[0001] This application relates to the field of speech processing technology, and in particular to a method, system, electronic device, and computer-readable storage medium for automatically generating rap music. Background Technology

[0002] With the development of hip-hop culture, rap music has become increasingly popular in recent years, attracting many people who are eager to try their hand at it, drawn by its youthful, energetic, and cool qualities. However, for ordinary users, the rhythm, cadence, and intonation of rap are difficult to control. Therefore, automated rapping technology can enable every rap enthusiast to rap, while also providing inspiration and assistance to professional rappers in their creative process.

[0003] However, current automated rapping technology cannot perfectly match the rhythm and cadence of music, and it lacks rhyming ability, so there is still a big gap between it and human-created rap music. Summary of the Invention

[0004] The main purpose of this application is to propose a method, system, electronic device, and computer-readable storage medium for automatically generating rap music, aiming to solve the problem of how to improve the effect of automatically generated rap music.

[0005] To achieve the above objectives, embodiments of this application provide a method for automatically generating rap music, the method comprising:

[0006] The input text is obtained, and the sentence and word segmentation results are obtained based on sentence and word segmentation technology;

[0007] Based on the rhythm detection results of the accompaniment music and the sentence segmentation results, words are filled in for each phrase of the accompaniment music;

[0008] Based on the lyrics, speech synthesis technology is used to obtain the rap audio of each sentence;

[0009] Each rap audio segment is spliced ​​together and mixed with the accompanying music to output the final rap result.

[0010] Optionally, the method further includes, after obtaining the input text:

[0011] The text is converted to phoneme level, and rhymes are determined based on statistical results for each phoneme; and

[0012] The sentence and word segmentation results of the text are obtained based on the rhyme scheme and sentence segmentation techniques.

[0013] Optionally, determining the rhyme based on the statistical results of each phoneme includes:

[0014] Calculate the frequency of each phoneme in the text;

[0015] Candidate rhymes are obtained based on the frequency;

[0016] The target rhyme is determined from the candidate rhymes.

[0017] Optionally, obtaining the sentence and word segmentation results of the text based on the rhyme and sentence segmentation techniques includes:

[0018] By combining the rhyme scheme and sentence segmentation techniques, along with a preset word count limit, the sentence segmentation result of the text is obtained;

[0019] Each sentence is segmented using word segmentation technology to obtain the segmentation results.

[0020] Optionally, the step of filling in words for each phrase of the accompaniment music based on the rhythm detection result and the phrase segmentation result includes:

[0021] Acquire the accompaniment music, perform beat detection on the accompaniment music, and obtain the accent and beat information;

[0022] Obtain each musical phrase of the accompaniment music and the corresponding accent and rhythm information, and fill in words for each musical phrase according to the number of characters in each phrase in the phrase segmentation and word segmentation results;

[0023] Record the position of each phrase and each word in the accompanying music after the lyrics are written.

[0024] Optionally, the step of filling in words for each musical phrase based on the number of characters in each phrase of the phrase segmentation result includes:

[0025] When the first participle in a clause is a singular word, start from the offbeat when filling in the blank.

[0026] Optionally, the step of filling in words for each musical phrase based on the number of characters in each phrase of the phrase segmentation result includes:

[0027] The three-character word segmentation is set as a triplete.

[0028] Optionally, the step of filling in words for each musical phrase based on the number of characters in each phrase of the phrase segmentation result includes:

[0029] If the number of characters in a clause exceeds a first preset number of characters, add complex rhythmic patterns to the multi-character words in the clause.

[0030] Optionally, the step of filling in words for each musical phrase based on the number of characters in each phrase of the phrase segmentation result includes:

[0031] If the number of characters in the phrase exceeds the second preset number of characters, borrow a beat from the previous musical phrase.

[0032] Optionally, the step of obtaining the rap audio of each sentence using speech synthesis technology based on the word filling results includes:

[0033] The duration of each word in each phrase is determined based on the lyrics, as well as the starting position of each phrase based on the musical phrase.

[0034] Based on the duration of each word and the start position of each clause, speech synthesis technology is used to convert each clause in the text into rap audio of corresponding duration.

[0035] Optionally, the method further includes the following before outputting the final rap result:

[0036] Enhance the speech in the mixing result.

[0037] Furthermore, to achieve the above objectives, embodiments of this application also provide an automatic rap music generation system, the system comprising:

[0038] The sentence segmentation module is used to acquire the input text and obtain the sentence segmentation and word segmentation results of the text based on sentence segmentation and word segmentation technology;

[0039] The word-filling module is used to fill in words for each phrase of the accompaniment music based on the rhythm detection results of the accompaniment music and the phrase segmentation results;

[0040] The synthesis module is used to obtain the rap audio of each sentence based on the word filling results using speech synthesis technology;

[0041] The mixing module is used to splice each of the rap voice audios and mix them with the accompaniment music to output the final rap result.

[0042] To achieve the above objectives, this application also provides an electronic device, which includes: a memory, a processor, and a rap music automatic generation program stored in the memory and executable on the processor. When the rap music automatic generation program is executed by the processor, it implements the rap music automatic generation method as described above.

[0043] To achieve the above objectives, embodiments of this application also provide a computer-readable storage medium storing a rap music automatic generation program, which, when executed by a processor, implements the rap music automatic generation method as described above.

[0044] The rap music automatic generation method, system, electronic device, and computer-readable storage medium proposed in this application can combine rhyme recognition technology, word segmentation and sentence segmentation technology, music timing technology, automatic flow algorithm, and speech synthesis technology to automatically generate perfectly matching rap music based on the input text and accompaniment music, and achieve automatic rhyming of sentences. Attached Figure Description

[0045] Figure 1 An application environment architecture diagram for implementing various embodiments of this application;

[0046] Figure 2 This is a flowchart of a method for automatically generating rap music according to the first embodiment of this application;

[0047] Figure 3 for Figure 2 Detailed flowchart of step S202;

[0048] Figure 4 for Figure 2 A detailed flowchart of step S204;

[0049] Figure 5 This is a schematic diagram of some commonly used rhythm patterns in the first embodiment of this application;

[0050] Figure 6 for Figure 2 A detailed flowchart of step S206;

[0051] Figure 7 for Figure 2 Detailed flowchart of step S208;

[0052] Figure 8 This is a flowchart of a method for automatically generating rap music according to the second embodiment of this application;

[0053] Figure 9 for Figure 8 Detailed flowchart of step S300;

[0054] Figure 10 for Figure 8 A detailed flowchart of step S302;

[0055] Figure 11 This is a flowchart illustrating another form of the automatic rap music generation method according to the second embodiment of this application;

[0056] Figure 12 This is a schematic diagram of the hardware architecture of an electronic device according to the third embodiment of this application;

[0057] Figure 13This is a schematic diagram of a rap music automatic generation system proposed in the fourth embodiment of this application;

[0058] Figure 14 This is a schematic diagram of a rap music automatic generation system proposed in the fifth embodiment of this application. Detailed Implementation

[0059] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application. All other embodiments obtained by those skilled in the art based on the embodiments in this application without inventive effort are within the scope of protection of this application.

[0060] It should be noted that the descriptions involving "first," "second," etc., in the embodiments of this application are for descriptive purposes only and should not be construed as indicating or implying their relative importance or implicitly specifying the number of technical features indicated. Therefore, a feature defined with "first" or "second" may explicitly or implicitly include at least one of that feature. Furthermore, the technical solutions of the various embodiments can be combined with each other, but this must be based on the ability of those skilled in the art to implement them. If the combination of technical solutions is contradictory or impossible to implement, it should be considered that such a combination of technical solutions does not exist and is not within the scope of protection claimed in this application.

[0061] Please see Figure 1 , Figure 1 This is an application environment architecture diagram for implementing various embodiments of this application. This application can be applied to application environments including, but not limited to, client 2, server 4, and network 6.

[0062] The client 2 is used to input text and background music, and to output the final rap result. The client 2 can be a terminal device such as a PC (Personal Computer), mobile phone, tablet computer, portable computer, or wearable device.

[0063] The server 4 is used to determine candidate rhymes and segment phrases / words based on the text input by the client 2. It fills in words into each musical phrase based on the rhythm detection results of the accompaniment music and the segmentation results. It uses speech synthesis technology to convert the text of each phrase into rap audio. Finally, it splices each audio segment and mixes it with the accompaniment music to obtain the final rap result. The server 4 can be a rack server, blade server, tower server, or cabinet server, etc., and can be a standalone server or a server cluster composed of multiple servers.

[0064] The network 6 can be an intranet, the Internet, Global System for Mobile Communication (GSM), Wideband Code Division Multiple Access (WCDMA), 4G network, 5G network, Bluetooth, Wi-Fi, or other wireless or wired networks. The server 4 and one or more clients 2 communicate via the network 6 to perform data transmission and interaction.

[0065] It is worth noting that the client 2 and the server 4 can also be integrated into a single electronic device, where input, output, and data processing are completed. Additionally, the application environment may include other necessary electronic devices, such as audio equipment.

[0066] Example 1

[0067] like Figure 2 The diagram shown is a flowchart of a method for automatically generating rap music according to the first embodiment of this application. It should be understood that the flowchart in this method embodiment is not intended to limit the order of execution steps. As needed, some steps in the flowchart can be added to or removed. The method will now be described using the server as the execution subject.

[0068] The method includes the following steps:

[0069] S202, Obtain the input text and obtain the sentence and word segmentation results of the text according to the sentence and word segmentation technology.

[0070] In this embodiment, users can input their own lyrics on the client, or input text obtained from other sources, or select the desired text from a list of alternative texts provided by the system. The server then segments the text into sentences and words using sentence and word segmentation technology to obtain the segmentation results.

[0071] For more details, please refer to the following: Figure 3 This is a detailed flowchart illustrating step S202 above. It is understood that this flowchart is not intended to limit the order of execution steps. Steps in this flowchart can be added to or removed as needed. In this embodiment, step S202 specifically includes:

[0072] S2020: Based on the sentence segmentation technology and the preset word count limit, the sentence segmentation result of the text is obtained.

[0073] In this embodiment, the text is segmented into sentences using sentence segmentation techniques and with a word limit applied, resulting in sentence segments suitable for rap. Specifically, the sentence segmentation can be performed using a deep learning framework based on ALBERT (multi-label text classification), BiLSTM (bidirectional long short-term memory network), and CRF (conditional random field). Each sentence must not exceed the word limit.

[0074] S2022 uses word segmentation technology to segment each sentence and obtain the segmentation results.

[0075] To ensure the semantic accuracy of the rap results, each sentence needs to be segmented into words after the sentence segmentation results are obtained. Word segmentation technology belongs to the field of natural language processing technology. Word segmentation is the technology of dividing words into segments in a sentence. The process of enabling computers to understand and segment words is called word segmentation.

[0076] Optionally, this embodiment can use jieba (a Python library specifically for word segmentation) to implement word segmentation. Its main word segmentation algorithm is a statistical one. Specifically, it includes: first, performing efficient word graph scanning based on a prefix dictionary to generate a directed acyclic graph (DAG) representing all possible word combinations for Chinese characters in a sentence; then, using dynamic programming to find the maximum probability path and identify the maximum segmentation combination based on word frequency; and for out-of-vocabulary (OV) words, using an HMM (Hidden Markov Model) based on the word-forming ability of Chinese characters, employing the Viterbi algorithm.

[0077] Back Figure 2 S204, Based on the rhythm detection results of the accompaniment music and the sentence segmentation results, fill in the lyrics for each phrase of the accompaniment music.

[0078] For more details, please refer to the following: Figure 4 This is a detailed flowchart illustrating step S204 above. It is understood that this flowchart is not intended to limit the order of execution steps. Steps in this flowchart can be added to or removed as needed. In this embodiment, step S204 specifically includes:

[0079] S2040, acquire the accompaniment music, perform beat detection on the accompaniment music, and obtain the accent and beat information.

[0080] Similar to the text above, users can enter their own original accompaniment music on the client, or enter accompaniment music obtained from other places, or select the desired accompaniment music from the music options provided by the system.

[0081] Once the accompaniment music is acquired, it is first subjected to beat tracking. Beat tracking is a music beat information detection technology that can determine the timing of musical downbeats and beats.

[0082] In this embodiment, any feasible beat detection technology can be used to detect the beat of the accompaniment music to obtain the corresponding accent and beat information, and no restrictions are imposed here.

[0083] Existing deep learning-based beat detection techniques generally consist of three steps: feature extraction, deep model beat probability estimation, and global beat position estimation. Feature extraction typically uses frequency domain features. The deep model generally employs sequence modeling networks such as BiLSTM. Considering the diversity of music genres, some algorithms fuse multiple models to model different types of music. The deep model can calculate the probability of each frame belonging to a beat point. The global beat position estimation combines features from traditional algorithms using dynamic programming-like algorithms, such as DBN (Deep Belief Network), to obtain a set of globally optimal beat positions.

[0084] S2042, obtain each musical phrase of the accompaniment music and the corresponding accent and rhythm information, and fill in words for each musical phrase according to the number of characters in each phrase in the phrase segmentation result.

[0085] This embodiment provides an automatic flow algorithm for lyric filling. The flow is a combination of accompaniment and lyrics, referring to the way words are pronounced during rapping, and the pronunciation or characteristics of each syllable. During rapping, it's necessary to handle the stress and unstress relationships between syllables, and the relationship between each syllable and the accompaniment. Technically, the flow is independent of the lyrics and accompaniment. In many cases, different rappers will handle the same lyrics very differently. The rhythm arrangement of the flow is generally centered around the beat. First, a beat is obtained, and then lyrics are filled into each beat of the beat, thus creating many different rhythmic patterns. The beat in rap is equivalent to the arrangement, a accompaniment created by the producer using drum beats, synthesizers, samples, etc. The beat is arguably the most important part of rapping.

[0086] In this embodiment, words can be added to each musical phrase based on the beat detection results and the number of words in each phrase, ensuring that each word corresponds perfectly to the musical phrase. For example, in 4 / 4 time accompaniment music, a musical phrase has eight beats, and two words can be added to each beat, so a musical phrase can hold at least sixteen words. If the current phrase has fewer than sixteen words, an empty beat can be added. If the current phrase has more than sixteen words, a more complex rhythmic pattern can be added, that is, more words can be added to a single beat, such as three or four words. Figure 5The diagram shown is a schematic of some commonly used rhythm patterns in this embodiment.

[0087] Furthermore, certain rules can be used to ensure that semantic information is not corrupted and to guarantee the diversity of rhythmic patterns. These rules include, for example:

[0088] (1) If the first participle in a clause is a singular word, then the word should be filled in from the reverse beat.

[0089] (2) If a word consists of three characters, it is set as a triplets, meaning that the three characters occupy one-third of a beat.

[0090] (3) Try to add pauses between word segments.

[0091] (4) Try to add complex rhythmic patterns to word segments with multiple characters.

[0092] (5) Dots can be added to two-character words. The dot refers to a dot placed to the right of the note head, used to indicate that a note needs to be played for half its original duration.

[0093] (6) If there are too many words, you can borrow beats from the previous musical phrase. "Too many words" refers to a phrase exceeding the preset word count. In this case, optionally, you can borrow the last two beats of the previous musical phrase. For example, for an eight-beat phrase, if the preceding phrase has fewer words, you can use only six beats to fill in the lyrics, and use the last two beats and the eight beats of the following phrase—a total of ten beats—to fill in the lyrics for the longer phrase corresponding to the following phrase.

[0094] S2044 records the position of each phrase and each word in the accompanying music after the lyrics are written.

[0095] After the lyrics for each musical phrase are completed, the start position (time) of each phrase in the accompanying music, as well as the start and end positions of each word in each phrase in the accompanying music, can be determined. This positional information is recorded for subsequent speech synthesis, splicing, and mixing.

[0096] Back Figure 2 S206, Based on the lyrics, use speech synthesis technology to obtain the rap audio of each sentence.

[0097] For more details, please refer to the following: Figure 6 This is a detailed flowchart illustrating step S206 above. It is understood that this flowchart is not intended to limit the order of execution steps. Steps in this flowchart can be added to or removed as needed. In this embodiment, step S206 specifically includes:

[0098] S2060 determines the duration of each word in each phrase based on the lyrics, and the starting position of each phrase based on the musical phrase.

[0099] Based on the start and end positions of each character in the lyrics within the accompanying music, the duration of each character in each phrase can be determined. Additionally, the start position of each phrase within the accompanying music can be determined based on the lyrics, such as the position entering on an offbeat, the position entering on a beat, or the position borrowing from the previous phrase.

[0100] S2062, based on the duration of each word and the start position of each clause, use speech synthesis technology to convert each clause in the text into rap audio of corresponding duration.

[0101] The text-to-speech (TTS) technology is a technique that generates artificial speech through mechanical and electronic methods. It can convert input text into corresponding audio reading content through machine conversion, and generally output audio files in formats such as pcm, wav, mp3, and aac.

[0102] Currently, the more mainstream TTS models include FastSpeech (an acoustic model) + HifiGan (a vocoder), VITS (a high-expression speech synthesis model), etc. In this embodiment, any feasible TTS model can be used to synthesize the text into speech, obtaining the rap audio corresponding to each sentence; no restrictions are placed here. Each rap audio corresponds to one sentence, and its duration is the same as the sentence.

[0103] Back Figure 2 S208, each of the rap voice audio segments is spliced ​​together and mixed with the accompaniment music to output the final rap result.

[0104] For more details, please refer to the following: Figure 7 This is a detailed flowchart illustrating step S208 above. It is understood that this flowchart is not intended to limit the order of execution steps. Steps in this flowchart can be added to or removed as needed. In this embodiment, step S208 specifically includes:

[0105] S2080, each of the rap voice audio segments is spliced ​​together according to the start position of each sentence.

[0106] Based on the starting position of each phrase in the accompaniment music, all the rap audio generated from the text can be spliced ​​together to obtain a complete rap audio segment.

[0107] S2082, mix the spliced ​​rap voice audio with the accompanying music.

[0108] The complete rap audio obtained after splicing is added to the accompaniment music according to the starting position of the first phrase, and then mixed with the accompaniment music to generate the desired rap result. The rap result generated in this embodiment, after being processed by the aforementioned steps, ensures that the voice and accompaniment music are perfectly matched, and each phrase can automatically rhyme.

[0109] Optionally, step S208 may further include:

[0110] S2084 enhances the speech in the mixing result and outputs the final rap result.

[0111] In addition to the above mixing, music effects such as EQ (Equalize) and reverb can be added to the voice (human voice) in the mixing result to enhance the generated rap result and perform fade-in and fade-out processing to obtain the final mixing result.

[0112] The automatic rap music generation method proposed in this embodiment combines word and sentence segmentation technology, music beat matching technology, automatic flow creation algorithm, and speech synthesis technology to automatically generate perfectly matching rap music based on the input text and accompaniment music. Specifically, the music beat matching technology ensures perfect synchronization between the synthesized speech and the accompaniment music; the automatic flow algorithm adaptively adjusts the rap flow based on the word and sentence segmentation results and the rhythm of the accompaniment music, making the final generated rap music more closely resemble manually created rap music.

[0113] Example 2

[0114] like Figure 8 The diagram shown is a flowchart of a method for automatically generating rap music according to a second embodiment of this application. In the second embodiment, the method for automatically generating rap music, based on the first embodiment described above, further includes step S300. It is understood that the flowchart in this method embodiment is not intended to limit the order of execution steps. Depending on the needs, some steps in the flowchart can be added to or removed.

[0115] The method includes the following steps:

[0116] S300: Obtain the input text, convert the text into phoneme level, and determine the rhyme based on the statistical results of each phoneme.

[0117] In this embodiment, the user can input the lyrics text created by themselves, or input the text obtained from other places, or select the required text from the alternative texts provided by the system. The server converts the input text to the phoneme level and determines the rhyme based on the phoneme conversion result and statistical methods.

[0118] Specifically, further refer to Figure 9 , which is a detailed flowchart of step S300 above. It can be understood that this flowchart is not used to limit the order of execution steps. According to needs, some steps in this flowchart can also be added or deleted. In this embodiment, the step S300 specifically includes:

[0119] S3000, obtain the input text content and convert the text to the phoneme level.

[0120] After obtaining the input text content, first convert the text to the phoneme level to obtain each phoneme. The phoneme is the smallest speech unit divided according to the natural attributes of speech. Analyzing based on the pronunciation actions in a syllable, one action constitutes one phoneme. Phonemes are divided into two major categories: vowels and consonants. For example, the Chinese syllable "啊(ā)" has only one phoneme, "爱(ài)" has two phonemes, "代(dài)" has three phonemes, etc.

[0121] Currently, there are already many tools or models that can convert text content to the corresponding phonemes. In this embodiment, any feasible phoneme conversion tool can be used for the conversion to obtain the phoneme result corresponding to the text, and no limitation is made here.

[0122] S3002, count the frequency of each phoneme appearing in the text.

[0123] After obtaining the phoneme result corresponding to the text, use statistical techniques to count the frequency of each phoneme appearing in the text. The purpose of counting the frequency is to determine the rhyme, so as to optimize the clause splitting result and achieve automatic rhyming of clauses. The rhyme refers to the rhyming words at the end of a rhymed text (such as poems, lyrics, songs, odes, etc.). Since rhyming mainly refers to the finals, this step can only count the frequency of the phonemes corresponding to the finals.

[0124] S3004, obtain candidate rhymes based on the frequency.

[0125] After counting the frequency of each phoneme appearing in the text, one or several phonemes with the highest frequency can be selected to generate the candidate rhymes.

[0126] S3006, determine the target rhyme from the candidate rhymes.

[0127] In this embodiment, the candidate rhymes can be provided to the user for judgment and selection of the final target rhyme, and the user can also modify the rhymes provided by the system. Alternatively, the target rhyme can be automatically determined according to preset rules, or the most frequently occurring phoneme can be directly identified as the target rhyme. The preset rules can be based on whether the candidate rhymes appear in a certain regular pattern, such as appearing every few words. If so, the candidate rhyme is identified as the target rhyme.

[0128] Back Figure 8 S302, the sentence and word segmentation results of the text are obtained based on the rhyme and sentence segmentation techniques.

[0129] In this embodiment, to ensure that the resulting sentences possess rhyming ability, the rhyme scheme is first determined based on phoneme statistics before sentence segmentation. Therefore, the text can be segmented into sentences and words by combining the rhyme scheme and sentence segmentation techniques to obtain the segmented sentence and word segmentation results.

[0130] For more details, please refer to the following: Figure 10 This is a detailed flowchart illustrating step S302 above. It is understood that this flowchart is not intended to limit the order of execution steps. Steps in this flowchart can be added to or removed as needed. In this embodiment, step S302 specifically includes:

[0131] S3020, combining the rhyme and sentence segmentation techniques, as well as the preset word count limit, the sentence segmentation result of the text is obtained.

[0132] In this embodiment, the text is segmented into sentences by combining sentence segmentation techniques with a word count limit, and then by combining the results of rhyme recognition, to obtain sentence segmentation results for rap. Specifically, the sentence segmentation process can be performed based on deep learning frameworks such as ALBERT, BiLSTM, and CRF. The target rhyme is used as the end of each sentence, and each sentence cannot exceed the word count limit.

[0133] S3022, use word segmentation technology to segment each sentence and obtain the segmentation results.

[0134] To ensure the semantic accuracy of the rap results, each sentence needs to be segmented into words after the sentence segmentation results are obtained. Optionally, this embodiment can use jieba for word segmentation, which mainly uses a statistical word segmentation algorithm. Specifically, it includes: first, performing efficient word graph scanning based on a prefix dictionary to generate a directed acyclic graph consisting of all possible word combinations of Chinese characters in the sentence. Then, dynamic programming is used to find the maximum probability path and identify the maximum segmentation combination based on word frequency. For out-of-vocabulary words, an HMM model based on the word-forming ability of Chinese characters is used, employing the Viterbi algorithm.

[0135] Back Figure 8 S304, Based on the rhythm detection results of the accompaniment music and the sentence segmentation results, fill in the lyrics for each phrase of the accompaniment music.

[0136] First, the accompaniment music is acquired, and its rhythm is detected to obtain the accent and beat information. Then, each musical phrase of the accompaniment music and its corresponding accent and beat information are acquired, and words are filled into each musical phrase according to the number of characters in each phrase in the phrase segmentation results.

[0137] This embodiment provides an automatic flow algorithm for word filling functionality. In this embodiment, words can be filled into each musical phrase based on the beat detection results and the number of characters in each phrase, ensuring that each word matches the musical phrase.

[0138] They can correspond perfectly. For example, in 4 / 4 time accompaniment music, a phrase has eight beats, and each beat can hold two words, so a phrase can hold at least sixteen words. If the current phrase has fewer than sixteen words, you can consider adding a blank beat. If the current phrase has more than sixteen words, you can consider adding a more complex rhythmic pattern, that is, filling a single beat with more words, such as three or four words.

[0139] 0. Furthermore, certain rules can be used to ensure that semantic information is not corrupted and to guarantee the diversity of rhythmic patterns. These rules include, for example:

[0140] (1) If the first participle in a clause is a singular word, then the word should be filled in from the reverse beat.

[0141] (2) If a word consists of three characters, it is set as a triplets, meaning that the three characters occupy one-third of a beat.

[0142] 5(3) Try to add pauses between word segments.

[0143] (4) Try to add complex rhythmic patterns to word segments with multiple characters.

[0144] (5) Dots can be added to two-character words. The dot refers to a dot placed to the right of the note head, used to indicate that a note needs to be played for half its original duration.

[0145] (6) If there are too many words, you can borrow a beat from the previous musical phrase. "Too many words" means that the number of words in a single phrase exceeds the preset limit. In this case, optionally, you can borrow a beat from the previous phrase.

[0146] The last two beats of a musical phrase. For example, for an eight-beat phrase, if the preceding phrase has fewer words, six beats can be used to write the lyrics. The last two beats and the eight beats of the following phrase, a total of ten beats, can be used to write the lyrics for the longer phrase corresponding to the following phrase.

[0147] After completing the lyrics for each musical phrase, the starting position (time interval) of each phrase in the accompanying music, as well as the starting and ending positions of each word in each phrase in the accompanying music, can be determined.

[0148] To confirm this, record this location information for subsequent speech synthesis, splicing, and mixing.

[0149] S306: Based on the lyrics, speech synthesis technology is used to obtain the rap audio of each sentence.

[0150] First, the duration of each character in each clause and the starting position of each clause based on the lyrics are determined. Then, based on the duration of each character and the starting position of each clause, speech synthesis technology is used to convert each clause in the text into a rap audio file of corresponding duration. In this embodiment, any feasible TTS model can be used to synthesize the text to obtain the rap audio file corresponding to each clause; no limitation is imposed here. Each rap audio file corresponds to one clause and has the same duration as the clause.

[0151] S308, splice each of the rap voice audios, mix them with the accompaniment music, and output the final rap result.

[0152] Based on the starting position of each phrase in the accompaniment music, all the rap audio generated from the text can be spliced ​​together to obtain a complete rap audio segment. This spliced ​​complete rap audio segment is then added to the accompaniment music according to the starting position of the first phrase, and mixed with the accompaniment music to generate the desired rap result. The rap result generated in this embodiment, after processing through the aforementioned steps, ensures that the speech and accompaniment music perfectly match, and each phrase can automatically rhyme.

[0153] Optionally, the speech in the mixed result can also be enhanced to output the final rap result. That is, in addition to the above mixing, music effects such as EQ (Equalize) and reverb can be added to the speech (voice) in the mixed result to enhance the generated rap result and perform fade-in and fade-out processing to obtain the final mixed result.

[0154] like Figure 11The diagram shown is a flowchart illustrating another form of the automatic rap music generation method described in this embodiment. Figure 11 The specific processes for each step have been explained above and will not be repeated here.

[0155] The automatic rap music generation method proposed in this embodiment combines rhyme recognition technology, word and sentence segmentation technology, music timing technology, automatic flow creation algorithm, and speech synthesis technology to automatically generate perfectly matching rap music based on the input text and accompaniment music. Specifically, rhyme recognition technology optimizes sentence segmentation results, achieving automatic rhyming; music timing technology ensures perfect synchronization between the synthesized speech and the accompaniment music; and the automatic flow algorithm adaptively adjusts the rap flow based on sentence and word segmentation results and the rhythm of the accompaniment music, making the final generated rap music more closely resemble manually created rap music.

[0156] Example 3

[0157] like Figure 12 The diagram shown illustrates the hardware architecture of an electronic device 20 according to a third embodiment of this application. In this embodiment, the electronic device 20 may include, but is not limited to, a memory 21, a processor 22, and a network interface 23, which are interconnected via a system bus. It should be noted that... Figure 12 Only the electronic device 20 with components 21-23 is shown; however, it should be understood that implementation of all shown components is not required, and more or fewer components may be implemented alternatively. In this embodiment, the electronic device 20 may be the server or the client.

[0158] The memory 21 includes at least one type of readable storage medium, including flash memory, hard disk, multimedia card, card-type memory (e.g., SD or DX memory), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, magnetic disk, optical disk, etc. In some embodiments, the memory 21 may be an internal storage unit of the electronic device 20, such as the hard disk or memory of the electronic device 20. In other embodiments, the memory 21 may also be an external storage device of the electronic device 20, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, etc., equipped on the electronic device 20. Of course, the memory 21 may include both the internal storage unit and the external storage device of the electronic device 20. In this embodiment, the memory 21 is typically used to store the operating system and various application software installed on the electronic device 20, such as the program code of the rap music automatic generation system 60. In addition, the memory 21 can also be used to temporarily store various types of data that have been output or will be output.

[0159] In some embodiments, the processor 22 may be a central processing unit (CPU), controller, microcontroller, microprocessor, or other data processing chip. The processor 22 is typically used to control the overall operation of the electronic device 20. In this embodiment, the processor 22 is used to run program code stored in the memory 21 or process data, such as running the rap music automatic generation system 60.

[0160] The network interface 23 may include a wireless network interface or a wired network interface, which is typically used to establish a communication connection between the electronic device 20 and other electronic devices.

[0161] Example 4

[0162] like Figure 13 The diagram shown is a modular schematic of a rap music automatic generation system 60 according to the fourth embodiment of this application. The rap music automatic generation system 60 can be divided into one or more program modules. One or more program modules are stored in a storage medium and executed by one or more processors to complete the embodiments of this application. The program modules referred to in the embodiments of this application are a series of computer program instruction segments capable of performing specific functions. The following description will specifically introduce the functions of each program module in this embodiment.

[0163] In this embodiment, the rap music automatic generation system 60 includes:

[0164] Sentence segmentation module 602 is used to acquire the input text and obtain the sentence segmentation and word segmentation results of the text according to sentence segmentation and word segmentation technology.

[0165] In this embodiment, the user can input their own lyrics on the client, or input text obtained from other sources, or select the desired text from the alternative texts provided by the system. The sentence segmentation module 602 performs sentence and word segmentation on the text using sentence and word segmentation technology to obtain the sentence and word segmentation results.

[0166] First, the text is segmented into sentences using sentence segmentation techniques and with a character limit applied, resulting in sentence segments suitable for rap. Specifically, the sentence segmentation can be performed using deep learning frameworks such as ALBERT, BiLSTM, and CRF. Each sentence must not exceed the specified character limit.

[0167] Then, word segmentation technology is used to segment each sentence, yielding the segmentation results. To ensure the semantic information of the rap result is correct, each sentence needs to be segmented after obtaining the segmentation results. Optionally, this embodiment can use jieba to implement word segmentation, which mainly uses a statistical word segmentation algorithm.

[0168] The word-filling module 604 is used to fill in words for each phrase of the accompaniment music based on the rhythm detection results of the accompaniment music and the phrase segmentation results.

[0169] First, the accompaniment music is acquired, and the rhythm of the accompaniment music is detected to obtain the accent and rhythm information.

[0170] Similar to the text above, users can enter their own original accompaniment music on the client, or enter accompaniment music obtained from other places, or select the desired accompaniment music from the music options provided by the system.

[0171] Once the accompaniment music is acquired, a beat detection is first performed on it. Beat detection is a music beat information detection technology that determines the timing of strong beats and rhythmic beats. In this embodiment, any feasible beat detection technology can be used to perform beat detection on the accompaniment music to obtain the corresponding strong beats and rhythmic beat information; no limitation is imposed here.

[0172] Then, each musical phrase of the accompaniment music and its corresponding accent and rhythm information are obtained, and lyrics are filled into each musical phrase according to the number of characters in each phrase in the phrase segmentation results. This embodiment provides an automatic Flow algorithm to implement the lyric filling function. The rhythm arrangement of the Flow is generally based on the Beat. First, a Beat is obtained, and then lyrics are filled into each beat of the Beat, which can extend to many rhythm patterns.

[0173] In this embodiment, words can be added to each musical phrase based on the beat detection results and the number of words in each phrase, ensuring that each word corresponds perfectly to the musical phrase. For example, in 4 / 4 time accompaniment music, a musical phrase has eight beats, and two words can be added to each beat, so a musical phrase can hold at least sixteen words. If the current phrase has fewer than sixteen words, an empty beat can be added. If the current phrase has more than sixteen words, a more complex rhythmic pattern can be added, that is, more words can be added to a single beat, such as three or four words.

[0174] Furthermore, certain rules can be used to ensure that semantic information is not corrupted and to guarantee the diversity of rhythmic patterns. These rules include, for example:

[0175] (1) If the first participle in a clause is a singular word, then the word should be filled in from the reverse beat.

[0176] (2) If a word consists of three characters, it is set as a triplets, meaning that the three characters occupy one-third of a beat.

[0177] (3) Try to add pauses between word segments.

[0178] (4) Try to add complex rhythmic patterns to word segments with multiple characters.

[0179] (5) Dots can be added to two-character words. The dot refers to a dot placed to the right of the note head, used to indicate that a note needs to be played for half its original duration.

[0180] (6) If there are too many words, you can borrow beats from the previous musical phrase. "Too many words" refers to a phrase exceeding the preset word count. In this case, optionally, you can borrow the last two beats of the previous musical phrase. For example, for an eight-beat phrase, if the preceding phrase has fewer words, you can use only six beats to fill in the lyrics, and use the last two beats and the eight beats of the following phrase—a total of ten beats—to fill in the lyrics for the longer phrase corresponding to the following phrase.

[0181] Finally, record the position of each phrase and each word in the accompanying music after the lyrics are written.

[0182] After the lyrics for each musical phrase are completed, the start position (time) of each phrase in the accompanying music, as well as the start and end positions of each word in each phrase in the accompanying music, can be determined. This positional information is recorded for subsequent speech synthesis, splicing, and mixing.

[0183] The synthesis module 606 is used to obtain the rap audio of each sentence using speech synthesis technology based on the word filling results.

[0184] First, the duration of each word in each phrase and the starting position of each phrase based on the lyrics are determined. The duration of each word in each phrase can be determined based on the start and end positions of each word in the accompaniment music according to the lyrics. Additionally, the starting position of each phrase in the accompaniment music based on the accompaniment music can be obtained from the lyrics, such as the position entering on an offbeat, the position entering on a beat, or the position borrowing from the previous phrase.

[0185] Then, based on the duration of each word and the start position of each clause, speech synthesis technology is used to convert each clause in the text into a rap audio file of corresponding duration. In this embodiment, any feasible TTS model can be used to synthesize the text to obtain the rap audio file corresponding to each clause; no limitation is imposed here. Each rap audio file corresponds to one clause and has the same duration as the clause.

[0186] The mixing module 608 is used to splice each of the rap voice audios and mix them with the accompaniment music to output the final rap result.

[0187] First, based on the starting position of each phrase in the accompaniment music, all the rap audio generated from the text can be spliced ​​together to obtain a complete rap audio segment. Then, the spliced ​​complete rap audio segment is added to the accompaniment music according to the starting position of the first phrase, and mixed with the accompaniment music to generate the desired rap result.

[0188] Optionally, the speech in the mixed result can be enhanced to output the final rap result. In addition to the above mixing, music effects such as EQ equalizers and reverb can be added to the speech (voice) in the mixed result to enhance the generated rap result and perform fade-in and fade-out processing to obtain the final mixed result.

[0189] Example 5

[0190] like Figure 14The diagram shown is a schematic representation of the modules of an automatic rap music generation system 60 according to the fifth embodiment of this application. In this embodiment, the automatic rap music generation system 60 includes, in addition to the phrase segmentation module 602, lyric filling module 604, synthesis module 606, and mixing module 608 described in the fourth embodiment, a statistics module 600.

[0191] The statistics module 600 is used to convert the text into phoneme level and determine the rhyme based on the statistical results of each phoneme.

[0192] In this embodiment, after obtaining the text input by the user, the statistics module 600 converts the input text into phoneme level and determines the rhymes based on the phoneme conversion results and statistical methods.

[0193] Specifically, after obtaining the input text content, the text is first converted into phoneme level to obtain individual phonemes. Once the phoneme results corresponding to the text are obtained, statistical techniques are used to count the frequency of each phoneme in the text. The purpose of counting the frequency is to determine the rhyme scheme, thereby optimizing the sentence segmentation results and achieving automatic rhyming. Since rhyming mainly refers to vowels, only the frequency of phonemes corresponding to vowels needs to be counted.

[0194] After calculating the frequency of each phoneme in the text, the most frequent phonemes or several can be selected to generate candidate rhymes. These candidate rhymes can be provided to the user for judgment and selection of the final target rhyme, and the user can also modify the rhymes provided by the system. Alternatively, the target rhyme can be automatically determined according to preset rules, or the most frequent phoneme can be directly selected as the target rhyme. The preset rules can be based on whether the candidate rhymes appear in a certain regular pattern, such as appearing every few words. If so, the candidate rhyme is selected as the target rhyme.

[0195] The sentence segmentation module 602 is also used to obtain the sentence segmentation and word segmentation results of the text based on the rhyme and sentence segmentation and word segmentation technology.

[0196] In this embodiment, to ensure that the resulting sentences possess rhyming ability, the rhyme scheme is first determined based on phoneme statistics before sentence segmentation. Therefore, the text can be segmented into sentences and words by combining the rhyme scheme and sentence segmentation techniques to obtain the segmented sentence and word segmentation results.

[0197] First, the text is segmented into sentences using sentence segmentation technology with a character limit, and then combined with rhyme recognition results to obtain sentence segments suitable for rap. Each sentence ends with the target rhyme, and each sentence cannot exceed the character limit. Then, word segmentation technology is used to segment each sentence, yielding the word segmentation results.

[0198] The rap music automatic generation system proposed in this embodiment combines rhyme recognition technology, word and sentence segmentation technology, music timing technology, automatic flow creation algorithm, and speech synthesis technology to automatically generate perfectly matching rap music based on the input text and accompaniment music. Specifically, rhyme recognition technology optimizes sentence segmentation results, achieving automatic rhyming; music timing technology ensures perfect synchronization between the synthesized speech and the accompaniment music; and the automatic flow algorithm adaptively adjusts the rap flow based on sentence and word segmentation results and the rhythm of the accompaniment music, making the final generated rap music more closely resemble human-created rap music.

[0199] Example 6

[0200] This application also provides another embodiment, namely, a computer-readable storage medium storing a rap music automatic generation program, which can be executed by at least one processor to perform the steps of the rap music automatic generation method as described above.

[0201] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0202] The sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0203] Obviously, those skilled in the art should understand that the modules or steps of the embodiments of this application described above can be implemented using general-purpose computing devices. They can be centralized on a single computing device or distributed across a network of multiple computing devices. Optionally, they can be implemented using computer-executable program code, thereby storing them in a storage device for execution by a computing device. In some cases, the steps shown or described can be performed in a different order than those presented here, or they can be fabricated as separate integrated circuit modules, or multiple modules or steps can be fabricated as a single integrated circuit module. Thus, the embodiments of this application are not limited to any particular combination of hardware and software.

[0204] The above are merely preferred embodiments of the present application and do not limit the patent scope of the present application. Any equivalent structural or procedural transformations made using the description and drawings of the present application, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of the present application.

Claims

1. A method for automatically generating rap music, characterized by, The method comprises: acquiring an input text, and obtaining a sentence segmentation result of the text according to a sentence segmentation technology; filling words into each phrase of accompaniment music according to a beat detection result of the accompaniment music and the sentence segmentation result; obtaining a rap voice audio of each phrase by using a voice synthesis technology according to a filling word result; splicing each rap voice audio and mixing the rap voice audio with the accompaniment music to output a final rap result; after acquiring the input text, the method further comprises: converting the text into a phoneme level, and determining a foot according to a statistical result of each phoneme; and obtaining the sentence segmentation result of the text according to the foot and the sentence segmentation technology; the determination of the foot according to the statistical result of each phoneme comprises: counting a frequency of occurrence of each phoneme in the text; obtaining a candidate foot according to the frequency; and determining a target foot from the candidate foot.

2. The method of claim 1, wherein, the obtaining of the sentence segmentation result of the text according to the foot and the sentence segmentation technology comprises: obtaining a phrase result of the text in combination with the foot and the sentence segmentation technology and a preset word limit; performing word segmentation on each phrase by using a word segmentation technology to obtain a word segmentation result.

3. The method of claim 1, wherein, the filling of words into each phrase of the accompaniment music according to the beat detection result of the accompaniment music and the sentence segmentation result comprises: acquiring the accompaniment music, performing beat detection on the accompaniment music to obtain heavy beat and beat information; acquiring each phrase of the accompaniment music and corresponding heavy beat and beat information, and filling words into each phrase according to a word number of each phrase in the sentence segmentation result; recording positions of each phrase and each word in the accompaniment music after the filling of words.

4. The method of claim 3, wherein, the filling of words into each phrase of the accompaniment music according to the beat detection result of the accompaniment music and the sentence segmentation result comprises: for a case that a first word segmentation in a phrase is a singular word, entering from a reverse beat when filling words. the filling of words into each phrase of the accompaniment music according to the beat detection result of the accompaniment music and the sentence segmentation result comprises:

5. The method of claim 3, wherein, setting a three-word segmentation as a triple. the filling of words into each phrase of the accompaniment music according to the beat detection result of the accompaniment music and the sentence segmentation result comprises:

6. The method of claim 3, wherein, for a case that a word number of the phrase exceeds a first preset word number, adding a complex rhythm type to a multi-word segmentation in the phrase. the filling of words into each phrase of the accompaniment music according to the beat detection result of the accompaniment music and the sentence segmentation result comprises: for a case that the word number of the phrase exceeds a second preset word number, borrowing a beat from a previous phrase.

7. The method of claim 3, wherein, the obtaining of the rap voice audio of each phrase by using the voice synthesis technology according to the filling word result comprises: determining a time length of each word in each phrase and a starting position of each phrase based on a phrase according to the filling word result; 8. The method of claim 1, wherein, converting each phrase in the text into a rap voice audio with a corresponding time length by using the voice synthesis technology according to the time length of each word and the starting position of each phrase. before outputting the final rap result, the method further comprises: performing beautification processing on a voice in a mixed result.

9. The method of claim 1, wherein, the system comprises: a sentence module, configured to acquire an input text, and obtain a sentence segmentation result of the text according to a sentence segmentation technology; 10. A system for automatically generating rap music, the system comprising: ​ ​ The lyric filling module is configured to fill lyrics into each musical phrase of the accompaniment music according to the beat detection result of the accompaniment music and the phrase segmentation result of the text; The synthesis module is configured to obtain a rap voice audio of each phrase by using a voice synthesis technique according to the filled lyrics; The mixing module is configured to splice each rap voice audio and mix the rap voice audio with the accompaniment music to output a final rap result; The statistics module is configured to convert the text into a phoneme level and determine a foot according to a statistical result of each phoneme; The phrase segmentation module is configured to obtain the phrase segmentation result of the text according to the foot and a phrase segmentation technique; The method for determining the foot according to the statistical result of each phoneme includes: counting a frequency of each phoneme in the text; obtaining a candidate foot according to the frequency; and determining a target foot from the candidate foot.

11. An electronic device, comprising: The electronic device includes a memory, a processor, and a rap music automatic generation program stored on the memory and executable on the processor, and the rap music automatic generation program, when executed by the processor, implements the rap music automatic generation method of any one of claims 1 to 9.

12. A computer-readable storage medium, characterized in that, The computer readable storage medium stores a rap music automatic generation program, and the rap music automatic generation program, when executed by the processor, implements the rap music automatic generation method of any one of claims 1 to 9.

13. A computer program product comprising computer instructions, characterised in that, The computer instructions, when executed by the processor, implement the rap music automatic generation method of any one of claims 1 to 9.

Images