Rap audio and video synthesis method, system and device and readable storage medium

Abstract

The application provides a rap audio-video synthesis method, system, device and readable storage medium. The method provided by the application can synthesize a rap song matched with the timbre of a user according to audio data recorded by the user. The user does not need to rap, and only needs to normally emit sound to complete audio recording to synthesize the rap song. The method provided by the embodiment of the application does not have requirements on the sound duration recorded by the user. After the user completes audio recording, a rap single video with the timbre of the user can be generated without waiting for a long time. Meanwhile, the audio data recorded by the user can be inspected to remind the user to re-record the audio when the audio data recorded by the user is unqualified, so as to improve the efficiency of synthesizing the rap song matched with the timbre of the user. The rap song matched with the timbre of the user finally synthesized has high musicity and pleasantness, and has details such as rhythm and musicality.

Description

Technical Field

[0001] This application relates to the field of audio and video data processing technology, and in particular to a method, system, device and readable storage medium for synthesizing rap audio and video. Background Technology

[0002] In recent years, driven by rap variety shows, more and more young people have been attracted to rap, and many are keen on trendy entertainment and consumption related to rap. Young people's interest in learning rap is also surging. However, the technical difficulty of rap performance differs from ordinary singing. In practice, people have found that even if they are familiar with the lyrics of rap songs, it is difficult to perform a decent rap, yet they still yearn to personally sing their favorite rap songs.

[0003] Existing AI-powered song generation software that mimics a user's voice cannot satisfy the needs of users generating rap songs. The generated songs lack rap rhythm, flow, and other musical characteristics, sounding unnatural. In addition, current technology requires a considerable amount of time for users to synthesize a song. First, users must upload at least several complete songs as learning materials, and it generally takes several hours to complete the training and synthesize a new song. Summary of the Invention

[0004] This application aims to at least solve one of the aforementioned technical defects. In view of this, this application provides a method, system, device and readable storage medium for synthesizing rap audio and video, in order to solve the technical defect of difficulty in synthesizing rap audio and video songs in the prior art.

[0005] A method for synthesizing rap audio and video, comprising:

[0006] In response to the user's selection of a target synthesized text, the selected target synthesized text is determined;

[0007] Based on the target synthesized text selected by the user, in response to the user clicking the record button, the system begins recording the audio data created by the user, and after the user finishes recording, the system obtains the user's original recording data as the target audio to be recorded.

[0008] The target recorded audio is subjected to quality inspection to obtain the quality inspection score result of the target recorded audio;

[0009] Determine whether the quality inspection score of the target recorded audio meets the preset standard for synthesized rap audio and video;

[0010] If the quality inspection score of the target recorded audio reaches the preset standard for synthesized rap audio and video, then the user's voiceprint features corresponding to the target recorded audio are extracted.

[0011] The user's voiceprint features are converted with a preset rap song template to obtain the target conversion result;

[0012] The target conversion result is mixed with a preset accompaniment to obtain a target rap song that matches the user's voice.

[0013] Preferably, the method further includes:

[0014] If the quality inspection score of the target recorded audio does not meet the preset standard for synthesized rap audio and video, the user is reminded to re-record the audio data, and after the user completes the re-recording of the audio data, the user's re-recorded audio data is obtained as the target recorded audio.

[0015] Return to the operation of performing quality inspection on the recorded audio of the target.

[0016] Preferably, after obtaining the target rap song that matches the user's voice, the method further includes:

[0017] A background video sample is randomly selected from a preset background video library as the target background video, or the target background video is selected by the user from the preset background video library.

[0018] The target rap song and the target background video are merged to obtain the target rap music video.

[0019] Preferably, the process of creating the preset rap song template includes:

[0020] Collect rap song clips from the target rapper;

[0021] Separate the rap vocals, accompaniment, and lyrics from a segment of the target rapper's rap song;

[0022] The lyrics of the rap song are annotated to obtain the lyric annotation results of the target rapper's rap song segment;

[0023] The rap lyrics annotation results are processed according to a preset format to obtain a lyrics file corresponding to the rap lyrics annotation results;

[0024] Extract semantic features of the vocal audio from the rap song clips of the target rapper;

[0025] A rap song template is synthesized by combining the semantic features of the rap lyrics file with the vocal audio in the target rapper's rap song clips.

[0026] A rap audio-visual synthesis system, applied to any of the rap audio-visual synthesis methods described above, the system comprising: a client and a server;

[0027] in,

[0028] The client responds to the user's click to enter the AI ​​rap interface and click the record button, records the user's target audio based on the rap lyrics read by the user, and uploads the target audio to the server after the user clicks to end the recording.

[0029] The server uses an audio quality detection module to detect the quality of the user's target recorded audio, obtains a quality inspection score for the target recorded audio, and returns the quality inspection score to the client. Based on the preset standards for synthesizing rap audio and video, and combined with the quality inspection score, the server determines whether the target recorded audio meets the preset standards for synthesizing rap audio and video. If the target recorded audio meets the preset standards for synthesizing rap audio and video, the server sends the target recorded audio to the rap synthesis service module on the server.

[0030] After receiving the target recorded audio, the rap synthesis service module performs noise reduction and voiceprint extraction on the target recorded audio. After obtaining the voiceprint feature vector corresponding to the user, it reads the semantic posterior probability features and fundamental frequency features in the preset rap song template, and inputs the semantic posterior probability features and fundamental frequency features in the preset rap song template and the voiceprint feature vector corresponding to the user into the preset conversion model and vocoder for synthesis processing to obtain the target rap audio corresponding to the user's timbre. At the same time, the target rap audio and related video materials are returned to the client.

[0031] The client, based on the target rap audio and related video materials returned by the rap synthesis service module of the server, draws a rap audio sung in the user's voice and generates a target rap music video with the video materials returned by the server as the background.

[0032] Preferably, the system further includes:

[0033] If the server determines that the quality inspection score of the target recorded audio does not meet the preset standard for synthesized rap audio and video, it sends a feedback message to the client reminding the user to re-record the audio data.

[0034] The client receives feedback information from the server reminding the user to re-record audio data, and after the user completes the re-recording, it obtains the user's re-recorded audio data as the target audio and sends it to the server.

[0035] Preferably, the system also includes an operations backend.

[0036] The operation backend is used to collect target rap song fragments from target rappers, and use a backing vocal and instrumental separation tool to separate the rap vocals and backing vocals from the target rap song fragments. The rap lyrics in the target rap song fragments are marked, processed into lyrics files of a preset format as required, and the rap vocals and backing vocals from the target rap song fragments, as well as the lyrics files of the preset format, are sent to the server.

[0037] The server receives the rap vocals and accompaniment from the target rap song segment, as well as the lyrics file in the preset format. It then uses a preset speech posterior probability model to extract semantic features from the lyrics file in the preset format. These features are then used to combine the rap vocals and accompaniment from the target rap song segment to synthesize a rap song template. The preset speech posterior probability model is trained using the lyrics file of a trained rap song as training samples. It is trained using the speaker semantic features corresponding to the posterior probability of each speech category at each specific time frame in the lyrics file of the trained rap song as sample labels.

[0038] Preferably, the preset format is an audio file format or a text document file format.

[0039] A rap audio / video synthesis device includes: one or more processors, and a memory;

[0040] The memory stores computer-readable instructions, which, when executed by the one or more processors, implement the steps of the rap audio-visual synthesis method as described above.

[0041] A readable storage medium storing computer-readable instructions, which, when executed by one or more processors, cause the one or more processors to perform the steps of the rap audio-visual synthesis method as described above.

[0042] As can be seen from the above-described technical solutions, when a user needs to synthesize rap music with their own unique vocal characteristics, the method provided in this application embodiment can respond to the user's selection of a target synthesis text. After determining the target synthesis text selected by the user, it responds to the user's click of the record button based on the selected target synthesis text, starts recording the user's created audio data, and after the user completes the recording, obtains the user's original recording data as the target recorded audio; performs quality inspection on the target recorded audio to obtain the quality inspection score result of the target recorded audio; so as to determine the quality inspection score result of the target recorded audio. If the quality inspection score of the target recorded audio meets the preset standard for synthesized rap audio and video, then it means that the target recorded audio can be used to synthesize the rap music needed by the user. Therefore, after determining that the quality inspection score of the target recorded audio meets the preset standard for synthesized rap audio and video, the user's voiceprint features corresponding to the target recorded audio can be extracted. In order to convert the user's voiceprint features with the preset rap song template, the target conversion result can be obtained. Finally, the target conversion result is mixed with the preset accompaniment, thereby obtaining a target rap song that matches the user's timbre.

[0043] The method provided in this application embodiment can synthesize a rap song matching the user's timbre by processing the user's recorded audio data. The user does not need to know how to rap; the user only needs to be able to make a sound and complete the audio recording to synthesize the rap song. The method provided in this application embodiment does not require a long recording duration. After the user completes the audio recording, there is no need to wait a long time before generating a rap single video with the user's timbre. At the same time, the method can also perform quality checks on the user's recorded audio data so that if the user's recorded audio data is unqualified, the user can be reminded to re-record the audio, thereby improving the efficiency of synthesizing a rap song that matches the user's timbre. Furthermore, the final synthesized rap song that matches the user's timbre has high musicality and pleasantness, and has details such as rhythm and musicality. Attached Figure Description

[0044] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments provided by this application. For those skilled in the art, other drawings can be obtained based on these drawings without any creative effort.

[0045] Figure 1 This application provides a system architecture diagram for implementing rap audio and video synthesis.

[0046] Figure 2 This application provides a signaling flowchart for implementing a rap audio and video synthesis method.

[0047] Figure 3 A flowchart illustrating a method for synthesizing rap audio and video is provided in this embodiment of the application.

[0048] Figure 4 This is a hardware structure block diagram of a rap audio-visual synthesis device disclosed in an embodiment of this application. Detailed Implementation

[0049] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0050] Currently, generating an AI-generated song requires a significant amount of time and doesn't provide immediate feedback. It can only generate pop songs at present, failing to reproduce the rhythm, flow, and musicality of rap tracks.

[0051] Furthermore, existing technologies often require users to submit at least several fully sung songs before they can be used for learning and training analysis when collecting users' original audio materials. Moreover, after generating songs, existing technologies typically match background videos that are simple, fixed GIFs or static images, failing to match detailed, varied, and engaging long videos.

[0052] Given that most current rap audio-visual synthesis solutions are ill-suited to complex and ever-changing business needs, this applicant has researched a rap audio-visual synthesis solution. This method can synthesize a rap song matching the user's timbre simply by processing the user's recorded audio data. The user does not need to know how to rap; they only need to be able to produce sound and complete the audio recording to synthesize the rap song. The method provided in this application does not have a requirement for the duration of the user's recorded audio. After the user completes the audio recording, there is no long waiting time before a rap single video with the user's timbre is generated. At the same time, the method can also perform quality checks on the user's recorded audio data, so that if the user's recorded audio data is unqualified, the user can be reminded to re-record the audio, thereby improving the efficiency of synthesizing a rap song matching the user's timbre. Furthermore, the final synthesized rap song matching the user's timbre has high musicality and pleasantness, and possesses details such as rhythm and musicality.

[0053] The methods provided in this application can be used in a variety of general-purpose or special-purpose computing device environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor devices, distributed computing environments including any of the above devices, etc.

[0054] This application provides a method for synthesizing rap audio and video. This method can be applied to various music editing systems or music management systems, as well as to various computer terminals or smart terminals. The executing entity can be the processor or server of the computer terminal or smart terminal.

[0055] The following is combined Figure 1 This application presents a schematic diagram of a system architecture for synthesizing rap lyrics, as shown in the embodiments of this application. Figure 1 As shown, the system architecture can include: a client, a server, and an operations backend, based on... Figure 1 The system architecture shown is as follows. Figure 2 This paper illustrates an optional signaling flow for the rap audio-visual synthesis method provided in an embodiment of this application, with reference to... Figure 2 The interaction process between the client, the server, and the operation backend can be as follows:

[0056] The client can respond to the user's click to enter the AI ​​rap interface and click the record button, so that it can record the user's target audio based on the rap lyrics read by the user, and upload the target audio to the server after the user clicks to end the recording.

[0057] The server may include a printing detection module and a rap synthesis service module.

[0058] In practical applications, after the server receives the target recorded audio, it can use a sound quality detection module to detect the quality of the user's target recorded audio, thereby obtaining a quality inspection score. This score indicates whether the target recorded audio meets the preset standards for synthesizing rap audio and video. Therefore, after obtaining the score, the server can return it to the client. Simultaneously, the server can also determine whether the target recorded audio meets the preset standards for synthesizing rap audio and video, based on the score. If the target recorded audio meets these standards, it indicates that it can be used to synthesize rap audio and video. Therefore, after determining that the target recorded audio meets the preset standards, it can be sent to the server's rap synthesis service module.

[0059] After receiving the target recorded audio, the rap synthesis service module can perform noise reduction and voiceprint extraction on the target recorded audio. After obtaining the voiceprint feature vector corresponding to the user, it can read the semantic posterior probability features and fundamental frequency features in the preset rap song template. The semantic posterior probability features and fundamental frequency features in the preset rap song template, as well as the voiceprint feature vector corresponding to the user, can be input into a preset conversion model and vocoder for synthesis processing. This can produce the target rap audio corresponding to the user's timbre. At the same time, the target rap audio and related video materials can be returned to the client for the client to synthesize a rap song corresponding to the user's timbre.

[0060] After receiving the target rap audio and related video materials returned by the rap synthesis service module of the server, the client can further draw a rap audio sung by the user's voice based on the target rap audio and related video materials returned by the rap synthesis service module of the server, and can generate a target rap music video with the video materials returned by the server as the background.

[0061] If the server determines that the quality inspection score of the target recorded audio does not meet the preset standard for synthesized rap audio and video, it can send feedback information to the client reminding the user to re-record the audio data.

[0062] After receiving feedback information from the server reminding the user to re-record audio data, the client can analyze the feedback information and remind the user to re-record audio data. After the user completes the re-recording, the client can obtain the user's re-recorded audio data as the target audio and send it to the server.

[0063] The operation backend can collect target rap song fragments from target rappers, and can use a backing vocal and instrumental separation tool to separate the rap vocals and backing vocals from the target rap song fragments. It can also annotate the rap lyrics in the target rap song fragments, process them into a preset format lyrics file as required, and send the rap vocals and backing vocals from the target rap song fragments, as well as the preset format lyrics file, to the server.

[0064] The preset formats are audio file formats and text document file formats.

[0065] After receiving the rap vocals and accompaniment from the target rap song segment, along with the lyrics file in the preset format, the server can use a preset speech posterior probability model to extract semantic features from the lyrics file. These features can then be combined with the rap vocals and accompaniment from the target rap song segment to synthesize a rap song template. The preset speech posterior probability model can be trained using the lyrics file of a trained rap song as training samples, and using the speaker semantic features corresponding to the posterior probability of each speech category at each specific time frame in the lyrics file as sample labels.

[0066] The posterior probability of speech is a time-to-class matrix that represents the posterior probability of each speech class for each specific time frame of a sentence. The posterior probability of a single phoneme as a function of time is called the posterior trajectory.

[0067] In practical applications, automatic speech recognition systems can convert acoustic features into features that are irrelevant to the speaker.

[0068] As can be seen from the above-described technical solutions, the rap audio-visual synthesis system provided in this application embodiment can process the audio data recorded by the user to synthesize a rap song that matches the user's timbre. The user does not need to know how to rap; the user only needs to be able to make a sound and complete the audio recording to complete the synthesis of the rap song. The system provided in this application embodiment has no requirement on the duration of the user's recorded audio. After the user completes the audio recording, there is no need to wait a long time before generating a rap single video with the user's timbre. At the same time, it can also perform quality checks on the user's recorded audio data so that if the user's recorded audio data is unqualified, the user can be reminded to re-record the audio, thereby improving the efficiency of synthesizing a rap song that matches the user's timbre. Furthermore, the final synthesized rap song that matches the user's timbre has high musicality and pleasantness, and has details such as rhythm and musicality.

[0069] The rap audio synthesized by the rap audio and video synthesis system provided in this application can also be matched with different cool video backgrounds to synthesize rap music short films with rich video backgrounds. The video background of the synthesized rap music is no longer a simple looping GIF or static image, but a truly exciting video with content. The process of saving the synthesized rap music short film is a frame-by-frame drawing of music video, making the synthesized rap music short film more personalized and with better viewing effect.

[0070] The following is combined Figure 2 The above-described rap audio-visual synthesis system will be used to introduce the flow of the rap audio-visual synthesis method provided in the embodiments of this application, such as... Figure 2 As shown, the process may include the following steps:

[0071] Step S101: In response to the user's operation of selecting the target synthesized text, determine the target synthesized text selected by the user.

[0072] Specifically, in practical applications, when a user wants to record a rap song, they will usually first determine the lyrics of the rap song they want to record.

[0073] To meet the needs of different users, the synthesized text of lyrics from different rappers' songs is usually stored on the server in advance.

[0074] Users can choose different synthesized texts to record their own audio data according to their preferences.

[0075] Therefore, the synthesized text can include lyrics text corresponding to different types of rap songs.

[0076] When a user wants to record their own rap song, the method provided in this application embodiment can respond to the user's operation of selecting a target synthesized text, determine the target synthesized text selected by the user, so that the user's reading audio can be recorded based on the target synthesized text.

[0077] Step S102: Based on the target synthesized text selected by the user, respond to the user's click of the record button, start recording the audio data created by the user, and after the user finishes recording, obtain the user's original recording data as the target recorded audio.

[0078] Specifically, as described above, the method provided in this application embodiment can determine the target synthesized text, which is the lyrics text corresponding to the rap song that the user wants to record.

[0079] Therefore, after determining the target synthesized text, the system can respond to the user's click of the record button based on the target synthesized text selected by the user, start recording the audio data created by the user, and after the user finishes recording, obtain the user's original recording data as the target recorded audio, so that the rap song that the user wants to record can be synthesized based on the target recorded audio.

[0080] in,

[0081] The target recorded audio may include the user's vocal characteristics.

[0082] In practical applications, when recording audio data, users do not need to know how to rap. Users only need to be able to make sounds normally and complete the audio recording according to the target synthesized text.

[0083] For example, after a user selects a target synthesized text, the user can click the record button on the interface to start recording the user's audio data. After clicking the record button, the user can read the selected target synthesized text aloud. After the user finishes reading the target synthesized text, the user can click the stop recording button to end the recording of the user's audio data. This application can obtain and save the user's original recording data as the target recorded audio.

[0084] Step S103: Perform quality inspection on the target recorded audio to obtain the quality inspection score result of the target recorded audio.

[0085] Specifically, after obtaining the target recorded audio, if the user records background noise or does not follow the target synthesized text when recording the target recorded audio, the final generated target recorded audio will be difficult to synthesize into the rap song that the user wants to synthesize.

[0086] Therefore, after determining the target recorded audio, the target recorded audio can be subjected to quality inspection to obtain the quality inspection score result of the target recorded audio, so as to determine whether the target recorded audio meets the preset standard of synthesized rap song through the quality inspection score result corresponding to the target recorded audio.

[0087] The quality inspection score reflects the quality of the target recorded audio data; the higher the score, the higher the quality of the target recorded audio.

[0088] Step S104: Determine whether the quality inspection score of the target recorded audio meets the preset standard for synthesized rap audio and video.

[0089] Specifically, since the quality inspection score can reflect the quality of the target recorded audio, after determining the quality inspection score, it can be further determined whether the quality inspection score of the target recorded audio meets the preset standard for synthesized rap audio and video.

[0090] In practical applications, if the quality inspection score of the target recorded audio reaches the preset standard for synthesizing rap audio and video, it means that the target recorded audio can be used to synthesize the rap song that the user wants to synthesize, and then step S105 can be executed.

[0091] If the quality inspection score of the target recorded audio does not meet the preset standard for synthesized rap audio and video, it means that the target recorded audio cannot be used to synthesize the rap song that the user wants to synthesize, and then step S106 can be executed.

[0092] Step S105: Extract the user voiceprint features corresponding to the target recorded audio.

[0093] Specifically, as described above, when the quality inspection score of the target recorded audio reaches the preset standard for synthesizing rap audio and video, the target recorded video can be used to synthesize the rap song that the user wants to synthesize. Therefore, the user's voiceprint features corresponding to the target recorded audio can be extracted so that a rap song corresponding to the user's timbre features can be synthesized based on the user's voiceprint features corresponding to the target recorded audio.

[0094] in,

[0095] The user's voiceprint features can reflect the user's vocal characteristics. Determining these user voiceprint features can help synthesize rap songs that incorporate the user's vocal characteristics.

[0096] Step S106: Remind the user to re-record audio data, and after the user completes the re-recording of audio data, obtain the user's re-recorded audio data as the target recording audio.

[0097] Specifically, if the quality inspection score of the target recorded audio does not meet the preset standard for synthesizing rap audio and video, it means that the target recorded audio cannot be used to synthesize the rap song that the user wants to synthesize. In this case, the user can be reminded to re-record the audio data. After the user completes the re-recording of the audio data, the user's re-recorded audio data is obtained as the target recorded audio. At the same time, the operation of performing quality inspection on the target recorded audio is returned so that user-recorded audio data that meets the preset standard for synthesizing rap audio and video can be obtained.

[0098] Step S107: Convert the user's voiceprint features with a preset rap song template to obtain the target conversion result.

[0099] Specifically, as described above, the user's voiceprint features can reflect the user's vocal characteristics. Therefore, in practical applications, after determining the user's voiceprint features, they can be converted against a preset rap song template to obtain the target conversion result.

[0100] The preset rap song templates can be set by referring to different rap songs by rappers of different styles.

[0101] Step S108: Mix the target conversion result with a preset accompaniment to obtain a target rap song that matches the user's timbre.

[0102] Specifically, in practical applications, the target conversion result is simply a rap song template that matches the user's timbre characteristics, obtained by converting the user's voiceprint features with a preset rap song template.

[0103] A rap song template alone is not enough to create the desired rap song effect for the user. Therefore, after determining the target conversion result, the target conversion result can be mixed with a preset accompaniment to obtain a target rap song that matches the user's voice.

[0104] In practical applications, after a user completes audio recording, the user no longer needs to wait a long time to convert the recorded audio into the rap song they want, which can effectively improve the efficiency of synthesizing rap songs.

[0105] As can be seen from the above-described technical solutions, the rap audio-visual synthesis method provided in this application embodiment can synthesize a rap song matching the user's timbre by processing the user's recorded audio data. The user does not need to know how to rap; the user only needs to be able to make a sound and complete the audio recording to complete the synthesis of the rap song. The system provided in this application embodiment has no requirement on the duration of the user's recorded audio. After the user completes the audio recording, there is no need to wait a long time before generating a rap single video with the user's timbre. At the same time, the system can also perform quality checks on the user's recorded audio data so that if the user's recorded audio data is unqualified, the user can be reminded to re-record the audio, thereby improving the efficiency of synthesizing a rap song that matches the user's timbre. Furthermore, the final synthesized rap song that matches the user's timbre has high musicality and pleasantness, and has details such as rhythm and musicality.

[0106] The rap audio synthesized by the method provided in this application embodiment can also be matched with different cool video backgrounds to synthesize rap music short films with rich video backgrounds. The video background of the synthesized rap music is no longer a simple looping GIF or static image, but a truly exciting video with content. The process of saving the synthesized rap music short film is a frame-by-frame drawing of music video, making the synthesized rap music short film more personalized and with better viewing effect.

[0107] In practical applications, after obtaining a target rap song that matches the user's voice, in order to obtain a better and more personalized rap song, the method provided in this application embodiment can also configure different background videos for the synthesized rap song to improve the viewing experience. The process is described below, and it may include the following steps:

[0108] Step S201: Randomly select a background video sample from the preset background video library as the target background video, or use the background video sample selected by the user from the preset background video library as the target background video.

[0109] Specifically, in practical applications, different users have different preferences. After obtaining a target rap song that matches the user's timbre, the method provided in this application embodiment can randomly determine a background video sample from a preset background video library as the target background video, or use a background video sample determined by the user from the preset background video library as the target background video, so as to combine the target rap song with the target background video to construct a rap music video.

[0110] For example, after determining a target rap song that matches the user's voice, the method provided in this application embodiment can randomly select a background video sample from a preset background video library for the user to choose from. If the user agrees to use a background video sample randomly selected from the preset background video library, then a background video sample randomly selected from the preset background video library will be used as the target background video. If the user does not agree to use a background video sample randomly selected from the preset background video library, then the background video sample determined by the user from the preset background video library can be used as the target background video.

[0111] Step S202: Merge the target rap song with the target background video to obtain the target rap music video.

[0112] Specifically, in practical applications, after determining the target background video, it is explained that the target background video can be used to create a rap music video. Therefore, the target rap song can be merged with the target background video to obtain the target rap music video.

[0113] As can be seen from the above-described technical solution, after obtaining a target rap song that matches the user's timbre, the method provided in this application embodiment can randomly determine a background video sample from a preset background video library as the target background video, or use a background video sample determined by the user from the preset background video library as the target background video, and merge the target rap song with the target background video to obtain a target rap music video, making the synthesized rap music video product more personalized and with better viewing effect.

[0114] As described above, the method provided in this application embodiment can create different rap song templates for users to choose from. The following describes the creation process of the preset rap song template, which may include the following steps:

[0115] Step S301: Collect rap song clips from the target rapper.

[0116] Specifically, in practical applications, when it is necessary to synthesize templates for rap songs of different styles, the method provided in this application embodiment can collect rap song fragments of the target rapper so that a rap song template similar to the target rapper's rap song can be synthesized based on the rap song fragments of the target rapper.

[0117] The method provided in this application does not require collecting a large number of complete rap songs from the target rapper. Instead, it can collect fragments of the target rapper's rap songs to create a rap song template similar in style to the target rapper's rap songs. This can effectively save time in synthesizing rap song templates and improve the efficiency of synthesizing rap song templates.

[0118] Step S302: Separate the rap vocals, rap accompaniment, and rap lyrics from the target rapper's rap song segment.

[0119] Specifically, in practical applications, after obtaining the rap song fragments of the target rapper, the rap song fragments can be further processed to form rap song templates.

[0120] For example, the rap vocals, accompaniment, and lyrics can be separated from a rap song segment by the target rapper.

[0121] Step S303: Annotate the rap lyrics to obtain the lyric annotation results of the target rapper's rap song segment.

[0122] Specifically, after obtaining the lyrics of the target rapper's rap song segment, in order to better match the rapper's vocal audio with the rap lyrics, the rap lyrics can be annotated to obtain the lyric annotation result of the target rapper's rap song segment, so that a rap song template can be synthesized based on the rap lyrics of the target rapper's rap song segment.

[0123] Step S304: Process the rap lyrics annotation results according to a preset format to obtain a lyrics file corresponding to the rap lyrics annotation results.

[0124] Specifically, after obtaining the rap lyrics, in order to better extract the vocal features of the target rapper based on the rap lyrics, the rap lyrics annotation results can be processed according to a preset format to obtain a lyrics file corresponding to the rap lyrics annotation results.

[0125] in,

[0126] Processing the rap song lyrics annotation results according to a preset format can help to use deep learning models to generate rap song templates in batches, thereby improving the efficiency and quantity of rap song template generation.

[0127] The preset format can be an audio file format or a text document file format.

[0128] Step S305: Extract the semantic features of the vocal audio from the target rapper's rap song clip.

[0129] Specifically, in practical applications, after separating the rap vocals, accompaniment, and lyrics from the target rapper's rap song segment, semantic features of the vocal audio in the target rapper's rap song segment can also be extracted.

[0130] The semantic features of the vocal audio in the target rapper's rap song clips can well interpret the rapper's rap song style. Determining the semantic features of the vocal audio in the target rapper's rap song clips can help generate rap song templates with similar semantic features to those in the target rapper's rap song clips.

[0131] Step S306: Combine the semantic features of the rap lyrics file with the vocal audio in the target rapper's rap song segment to synthesize a rap song template.

[0132] Specifically, in practical applications, the semantic features of the vocal audio in the target rapper's rap song segment can reflect the rapper's rapping style and characteristics. Based on the semantic features of the vocal audio in the target rapper's rap song segment and the rap lyrics file, a rap song template similar to the target rapper's rap song can be generated.

[0133] Therefore, after determining the semantic features of the vocal audio in the rap lyrics file and the target rapper's rap song segment, a rap song template can be synthesized by combining the semantic features of the rap lyrics file and the vocal audio in the target rapper's rap song segment.

[0134] As can be seen from the above-described technical solutions, the method provided in this application embodiment can collect rap song fragments from target rappers; separate the rap vocals, rap accompaniment, and rap lyrics from the target rapper's rap song fragments; after obtaining the rap lyrics, further annotate the rap lyrics to obtain the lyric annotation result of the target rapper's rap song fragments; process the rap lyric annotation result according to a preset format to obtain a lyric file corresponding to the rap lyric annotation result; it can also extract the semantic features of the vocal audio in the target rapper's rap song fragments; finally, it can combine the rap lyric file with the semantic features of the vocal audio in the target rapper's rap song fragments to synthesize a rap song template. The method provided in this application embodiment can quickly and in batches synthesize rap song templates of different rap styles for different users to choose from, providing users with diversified rap song templates, and helping to improve the efficiency of synthesizing the rap songs that users need.

[0135] The specific processing flow included in the above-mentioned rap audio and video synthesis method can be found in the relevant introduction of the rap audio and video synthesis system section above, and will not be repeated here.

[0136] Furthermore, in practical applications, a corresponding rap audio-visual device that can implement the processing procedure of the rap audio-visual method provided in the embodiments of this application can also be generated based on the rap audio-visual synthesis method provided in the embodiments of this application.

[0137] The rap audio-visual synthesis device generated by the rap audio-visual synthesis method provided in the embodiments of this application can be applied to rap audio-visual synthesis equipment, such as terminals: mobile phones, computers, etc. Optionally, Figure 3 The hardware structure block diagram of the rap audio and video synthesis device is shown, with reference to... Figure 3 The hardware structure of a rap audio and video synthesis device may include: at least one processor 1, at least one communication interface 2, at least one memory 3, and at least one communication bus 4.

[0138] In this embodiment, the number of processor 1, communication interface 2, memory 3, and communication bus 4 is at least one, and processor 1, communication interface 2, and memory 3 communicate with each other through communication bus 4.

[0139] Processor 1 may be a central processing unit (CPU), an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of this application.

[0140] Memory 3 may include high-speed RAM, and may also include non-volatile memory, such as at least one disk storage device;

[0141] The memory stores a program, which the processor can call. The program is used to implement the various processing steps in the aforementioned terminal rap audio-visual synthesis scheme.

[0142] This application embodiment also provides a readable storage medium that can store a program suitable for processor execution, the program being used to: implement the various processing flows of the aforementioned terminal in the rap audio and video synthesis scheme.

[0143] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0144] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.

[0145] The above description of the disclosed embodiments enables those skilled in the art to make or use this application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this application. Various embodiments can be combined with each other. Therefore, this application is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for rap audio-video synthesis, characterized in that, include: In response to the user's selection of a target synthesized text, the selected target synthesized text is determined; Based on the target synthesized text selected by the user, in response to the user clicking the record button, the system begins recording the audio data created by the user, and after the user finishes recording, the system obtains the user's original recording data as the target audio to be recorded. The target recorded audio is subjected to quality inspection to obtain the quality inspection score result of the target recorded audio; Determine whether the quality inspection score of the target recorded audio meets the preset standard for synthesized rap audio and video; If the quality inspection score of the target recorded audio reaches the preset standard for synthesized rap audio and video, then the user's voiceprint features corresponding to the target recorded audio are extracted. The user's voiceprint features are converted with a preset rap song template to obtain the target conversion result; Among them, the preset rap song template is a lyrics file generated from rap song fragments with rap lyrics annotation results, and a semantic feature synthesis of human voice audio in rap song fragments; The target conversion result is mixed with a preset accompaniment to obtain a target rap song that matches the user's timbre; The process of creating the preset rap song template includes: Collect rap song clips from the target rapper; Separate the rap vocals, accompaniment, and lyrics from a segment of the target rapper's rap song; The lyrics of the rap song are annotated to obtain the lyric annotation results of the target rapper's rap song segment; The rap lyrics annotation results are processed according to a preset format to obtain a lyrics file corresponding to the rap lyrics annotation results; Extract semantic features of the vocal audio from the rap song clips of the target rapper; A rap song template is synthesized by combining the semantic features of the rap lyrics file with the vocal audio in the target rapper's rap song clips.

2. The method of claim 1, wherein, The method also includes: If the quality inspection score of the target recorded audio does not meet the preset standard for synthesized rap audio and video, the user is reminded to re-record the audio data, and after the user completes the re-recording of the audio data, the user's re-recorded audio data is obtained as the target recorded audio. Return to the operation of performing quality inspection on the recorded audio of the target.

3. The method of claim 1, wherein, After obtaining the target rap song that matches the user's vocal tone, the method also includes: A background video sample is randomly selected from a preset background video library as the target background video, or the target background video is selected by the user from the preset background video library. The target rap song and the target background video are merged to obtain the target rap music video.

4. A rap audio-video synthesis system characterized by, The rap audio and video synthesis method applied to any one of claims 1-3 above, the system includes: a client, a server, and an operation backend; The client responds to the user's click to enter the AI ​​rap interface and click the record button, records the user's target audio based on the rap lyrics read by the user, and uploads the target audio to the server after the user clicks to end the recording. The server uses an audio quality detection module to detect the quality of the user's target recorded audio, obtains a quality inspection score for the target recorded audio, and returns the quality inspection score to the client. Based on the preset standards for synthesizing rap audio and video, and combined with the quality inspection score, the server determines whether the target recorded audio meets the preset standards for synthesizing rap audio and video. If the target recorded audio meets the preset standards for synthesizing rap audio and video, the server sends the target recorded audio to the rap synthesis service module on the server. After receiving the target recorded audio, the rap synthesis service module performs noise reduction and voiceprint extraction on the target recorded audio. After obtaining the voiceprint feature vector corresponding to the user, it reads the semantic posterior probability features and fundamental frequency features in the preset rap song template, and inputs the semantic posterior probability features and fundamental frequency features in the preset rap song template and the voiceprint feature vector corresponding to the user into the preset conversion model and vocoder for synthesis processing to obtain the target rap audio corresponding to the user's timbre. At the same time, the target rap audio and related video materials are returned to the client. Among them, the preset rap song template is a lyrics file generated from rap song fragments with rap lyrics annotation results, and a semantic feature synthesis of human voice audio in rap song fragments; The operation backend is used to collect rap song fragments of the target rapper, separate the rap vocals, rap accompaniment and rap lyrics in the rap song fragments of the target rapper, annotate the rap lyrics to obtain the lyrics annotation results of the rap song fragments of the target rapper, and process the rap lyrics annotation results according to a preset format to obtain the lyrics file corresponding to the rap lyrics annotation results; The server is also used to extract the semantic features of the human voice audio in the rap song segment of the target rapper, and combine the rap lyrics file with the semantic features of the human voice audio in the rap song segment of the target rapper to synthesize a rap song template. The client, based on the target rap audio and related video materials returned by the rap synthesis service module of the server, draws a rap audio sung in the user's voice and generates a target rap music video with the video materials returned by the server as the background.

5. The system of claim 4, wherein, The system also includes: If the server determines that the quality inspection score of the target recorded audio does not meet the preset standard for synthesized rap audio and video, it sends a feedback message to the client reminding the user to re-record the audio data. The client receives feedback information from the server reminding the user to re-record audio data, and after the user completes the re-recording, it obtains the user's re-recorded audio data as the target audio and sends it to the server.

6. The system of claim 4, wherein, The operation backend is used to collect target rap song fragments from target rappers, and use a backing vocal and instrumental separation tool to separate the rap vocals and backing vocals from the target rap song fragments. The rap lyrics in the target rap song fragments are marked, processed into lyrics files of a preset format as required, and the rap vocals and backing vocals from the target rap song fragments, as well as the lyrics files of the preset format, are sent to the server. The server receives the rap vocals and accompaniment from the target rap song segment, as well as the lyrics file in the preset format. It then uses a preset speech posterior probability model to extract semantic features from the lyrics file in the preset format. These features are then used to combine the rap vocals and accompaniment from the target rap song segment to synthesize a rap song template. The preset speech posterior probability model is trained using the lyrics file of a trained rap song as training samples. It is trained using the speaker semantic features corresponding to the posterior probability of each speech category at each specific time frame in the lyrics file of the trained rap song as sample labels.

7. The system of claim 6, wherein, The preset formats are audio file format and text document file format.

8. A rap audio-video synthesis device, characterized by, include: One or more processors, and memory; The memory stores computer-readable instructions, which, when executed by the one or more processors, implement the steps of the rap audio-visual synthesis method as described in any one of claims 1 to 3.

9. A readable storage medium characterized by: The readable storage medium stores computer-readable instructions that, when executed by one or more processors, cause the one or more processors to perform the steps of the rap audio-visual synthesis method as described in any one of claims 1 to 3.

Images