Method, computer device, storage medium and program product for generating a sung audio
Patent Information
- Application Number
- CN202611014085.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-07-08
- Publication Date
- 2026-10-30
AI Technical Summary
[0003]在目前的歌声转换技术中，用户的录音音频仅是提供了用户的音色信息，这样在生成的目标歌曲中仅能体现出用户的音色特点
在本申请提供的技术方案中，可以在用户实际演唱的目标歌曲的第一演唱音频中，提取蕴含咬字习惯、口音等特点的语义特征，以及用户的对应的音色特征。在歌手演唱目标歌曲的第二演唱音频中提取旋律特征，并根据发音单元分别在第一演唱音频和第二演唱音频中的时间信息，对旋律特征进行调整得到适配用户的目标旋律特征。这样基于专业歌手的目标旋律特征与用户的语义特征和音色特征生成的演唱音频中，即满足了对旋律准确性的要求，还可以体现用户的音色、咬字习惯、口音等特点，能够提高为用户生成的演唱音频的效果。
Smart Images
Figure CN123016961A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of artificial intelligence technology, and in particular to a method, computer device, storage medium, and program product for generating vocal audio. Background Technology
[0002] Singing Voice Conversion (SVC) is an AI-powered voice conversion technology that can transform one person's singing voice into another's. For example, it can convert the voice of a professional singer into the voice of an ordinary user, thereby increasing the flexibility and enjoyment of music playback.
[0003] Current vocal conversion technology only provides the user's timbre information in the user's recorded audio. Therefore, the generated target song only reflects the user's timbre characteristics. However, features such as the user's pronunciation habits and accent cannot be reflected in the generated singing audio, resulting in low quality. Summary of the Invention
[0004] This application provides a method, computer device, storage medium, and program product for generating vocal audio, which can improve the quality of the generated vocal audio. The corresponding technical solution is as follows: Firstly, a method for generating vocal audio is provided, the method comprising: Obtain the first audio recording of the user singing the target song and the second audio recording of the singer singing the target song; Extract the semantic features and timbre features corresponding to the first singing audio, and extract the melody features corresponding to the second singing audio; After aligning the first and second singing audios at the pronunciation unit level, the first time information corresponding to each pronunciation unit in the first singing audio and the second time information corresponding to each pronunciation unit in the second singing audio are determined. Based on the first time information and the second time information, determine the timing scaling parameters corresponding to each sound unit; Based on the timing scaling parameters corresponding to each vocal unit, the melodic features corresponding to each vocal unit are adjusted in timing to obtain the target melodic features; Based on the target melody features, the semantic features, and the timbre features, a third singing audio of the user singing the target song is generated.
[0005] Optionally, determining the timing scaling parameters corresponding to each vocal unit based on the first timing information and the second timing information includes: Based on the first time information and the second time information, determine the first pronunciation duration of each pronunciation unit in the first singing audio and the second pronunciation duration in the second singing audio; The ratio of the first pronunciation duration and the second pronunciation duration corresponding to each pronunciation unit is determined as the timing scaling parameter for each pronunciation unit.
[0006] Optionally, the first and second pronunciation durations corresponding to each pronunciation unit respectively include the pronunciation durations corresponding to the start period, the duration, and the end period. The ratio of the first and second pronunciation durations includes the first ratio corresponding to the start period, the second ratio corresponding to the duration, and the third ratio corresponding to the end period. The step of adjusting the melody features corresponding to each vocal unit according to the timing scaling parameters of each vocal unit includes: for each vocal unit, scaling the melody features corresponding to the starting time period according to the first ratio, scaling the melody features corresponding to the continuous time period according to the second ratio, and scaling the melody features corresponding to the ending time period according to the third ratio.
[0007] Optionally, the melody features corresponding to the pronunciation unit include the melody sub-features corresponding to a first number of audio frames in the second singing audio; The step of adjusting the melody features corresponding to each vocal unit according to the timing scaling parameter corresponding to each vocal unit to obtain the target melody features includes: for each vocal unit, determining the product of the first number of audio frames corresponding to the vocal unit in the second singing audio and the ratio to obtain the second number; For each vocal unit, the first number of melodic sub-features corresponding to the vocal unit are interpolated or pruned to obtain the target melodic feature composed of the second number of melodic sub-features.
[0008] Optionally, the pronunciation unit includes phonemes, syllables, or words.
[0009] Optionally, generating a third vocal audio recording of the user singing the target song based on the target melody features, the semantic features, and the timbre features includes: A reference timbre feature matching the timbre feature is determined in the timbre library, wherein the timbre library includes timbre features corresponding to multiple singers; a third singing audio of the user singing the target song is generated based on the target melody feature, the semantic feature, and the reference timbre feature.
[0010] Optionally, generating a third vocal audio recording of the user singing the target song based on the target melody features, the semantic features, and the reference timbre features includes: The timbre features are fused with the matched reference timbre features to obtain fused timbre features; based on the target melody features, the semantic features, and the fused timbre features, a third singing audio of the user singing the target song is generated.
[0011] Secondly, an apparatus for generating vocal audio is provided, the apparatus comprising: The acquisition module is used to acquire the first audio recording of the user singing the target song and the second audio recording of the singer singing the target song. The extraction module is used to extract the semantic features and timbre features corresponding to the first singing audio, and to extract the melody features corresponding to the second singing audio; The determining module is used to determine, after performing articulation unit-level alignment on the first singing audio and the second singing audio, the first timing information corresponding to each articulation unit in the first singing audio and the second timing information corresponding to each articulation unit in the second singing audio; and to determine the timing scaling parameter corresponding to each articulation unit based on the first timing information and the second timing information. The adjustment module is used to perform temporal adjustment on the melodic features corresponding to each vocal unit according to the temporal scaling parameters corresponding to each vocal unit, so as to obtain the target melodic features; The generation module is used to generate a third singing audio of the user singing the target song based on the target melody features, the semantic features, and the timbre features.
[0012] Optionally, the determining module is configured to: Based on the first time information and the second time information, determine the first pronunciation duration of each pronunciation unit in the first singing audio and the second pronunciation duration in the second singing audio; The ratio of the first pronunciation duration and the second pronunciation duration corresponding to each pronunciation unit is determined as the timing scaling parameter for each pronunciation unit.
[0013] Optionally, the first and second pronunciation durations corresponding to each pronunciation unit respectively include the pronunciation durations corresponding to the start period, the duration, and the end period. The ratio of the first and second pronunciation durations includes the first ratio corresponding to the start period, the second ratio corresponding to the duration, and the third ratio corresponding to the end period. The adjustment module is configured to: for each vocal unit, scale the melody features corresponding to the starting time segment according to the first ratio, scale the melody features corresponding to the duration segment according to the second ratio, and scale the melody features corresponding to the ending time segment according to the third ratio.
[0014] Optionally, the melody features corresponding to the pronunciation unit include the melody sub-features corresponding to a first number of audio frames in the second singing audio; The adjustment module is configured to: for each vocal unit, determine the product of a first number of audio frames corresponding to the vocal unit in the second singing audio and the ratio to obtain a second number; for each vocal unit, perform interpolation or deletion processing on the first number of melodic sub-features corresponding to the vocal unit to obtain the target melody feature composed of the second number of melody sub-features.
[0015] Optionally, the pronunciation unit includes phonemes, syllables, or words.
[0016] Optionally, a generation module is used to determine a reference timbre feature that matches the timbre feature in a timbre library, wherein the timbre library includes timbre features corresponding to multiple singers; and to generate a third singing audio of the user singing the target song based on the target melody feature, the semantic feature, and the reference timbre feature.
[0017] Optionally, the generation module is used to perform timbre fusion processing on the timbre features and the matched reference timbre features to obtain fused timbre features; and to generate a third singing audio of the user singing the target song based on the target melody features, the semantic features and the fused timbre features.
[0018] Thirdly, a computer device is provided, the computer device including a processor and a memory, the memory storing at least one instruction, the instruction being loaded and executed by the processor to perform the operation of generating vocal audio as described in the first aspect above.
[0019] Fourthly, a computer-readable storage medium is provided, the storage medium storing at least one instruction, the instruction being loaded and executed by a processor to perform the operations of generating vocal audio as described in the first aspect above.
[0020] Fifthly, a computer program product is provided, characterized in that the storage medium stores at least one piece of program code, which is loaded and executed by a processor to perform the operations of generating singing audio as described in the first aspect above.
[0021] The beneficial effects of the technical solution provided in this application are: The technical solution provided in this application can extract semantic features containing characteristics such as pronunciation habits and accents, as well as the user's corresponding timbre features, from the first audio recording of the target song actually sung by the user. Melodic features are extracted from the second audio recording of the target song sung by the singer, and these melody features are adjusted based on the time information of the pronunciation units in the first and second audio recordings to obtain target melody features adapted to the user. Thus, the singing audio generated based on the target melody features of a professional singer and the user's semantic and timbre features not only meets the requirement of melody accuracy but also reflects the user's timbre, pronunciation habits, accent, and other characteristics, thereby improving the quality of the singing audio generated for the user. Attached Figure Description
[0022] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0023] Figure 1 This is a schematic diagram of a traditional method for generating audio of a user singing a target song; Figure 2 This is a flowchart of a method for generating singing audio provided in an embodiment of this application; Figure 3 This is a schematic diagram of a method for generating singing audio provided in an embodiment of this application; Figure 4 This is a schematic diagram of a device for generating singing audio provided in an embodiment of this application; Figure 5 This is a schematic diagram of the structure of a computer device provided in an embodiment of this application; Figure 6 This is a schematic diagram of the structure of a computer device provided in an embodiment of this application. Detailed Implementation
[0024] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application will be described in further detail below with reference to the accompanying drawings.
[0025] Vocal synthesis technology is a speech synthesis technique designed to enable computers to produce beautiful and melodious singing voices, much like humans. As a new application area of speech synthesis technology, vocal synthesis has considerable application value and promising prospects in fields such as virtual singers, record production, and digital music creation.
[0026] Vocal synthesis technology utilizes artificial intelligence to generate audio of the user singing a target song, based on the user's timbre. The resulting audio sounds just like the actual singing of the user. Figure 1 This is a schematic diagram of a traditional method for generating audio of a user singing a target song. (Example) Figure 1 As shown, the process first acquires the introductory audio of the target song and the user's recorded audio. The target song can be the song the user wishes to generate a vocal audio recording for; the introductory audio can be the vocal audio of the target song performed by a professional singer; and the user's recorded audio can be audio of the user speaking or singing a song for a certain duration, captured from the user's terminal.
[0027] In one example, the terminal's running music application has an AI-generated singing audio function. Users can record or upload their own audio (i.e., record a song) within the corresponding interface and select the song they want to sing (i.e., the target song). The terminal can then send the recorded audio and the target song's identifier to the server, which will then execute... Figure 1 The method shown is for generating audio of a user singing a target song.
[0028] Understandably, the lead vocal audio provides the melody and semantic information for the vocal audio to be generated, while the user's recorded audio provides the user's timbre information. See also... Figure 1 The system can input the lead vocal audio into both the melody extractor and the semantic extractor, obtaining the melody features output by the melody extractor and the semantic features output by the semantic extractor. Similarly, the user's recorded audio can be input into the audio extractor to obtain the user's timbre features. Then, the melody features, semantic features, and timbre features can be input into the acoustic model to obtain the audio data output by the acoustic model. Finally, the vocoder decodes the audio data to obtain a performance audio of the target song generated according to the user's timbre and the professional singer's vocal skills.
[0029] It is evident that in the current process of generating singing samples for target songs, the user's recorded audio only provides their vocal timbre characteristics. Therefore, the generated target song only reflects these timbre features. However, the user's pronunciation habits, accent, and other characteristics cannot be reflected in the generated singing audio. It is understandable that the purpose of generating singing audio through vocal synthesis technology is to predict the singing quality a user would produce after receiving professional vocal training. However, if the generated singing audio lacks the user's pronunciation habits, accent, and other characteristics, it will lack realism, resulting in a lower quality of the currently generated singing audio.
[0030] This application provides a method for generating vocal audio. In this method, semantic features containing characteristics such as pronunciation habits and accents can be extracted from the actual vocal audio of a target song sung by a user. The user's corresponding timbre features are also extracted, and the melody features corresponding to the professional singer's introductory audio are further extracted. Thus, by combining the user's semantic features, timbre features, and the professional singer's melody features, the generated vocal audio for the target song can reflect the user's timbre, pronunciation habits, accent, and other characteristics, thereby improving the quality of the generated vocal audio.
[0031] Figure 2 This application provides a method for generating singing audio, which can be executed by a server, such as... Figure 2 As shown, the method includes: Step 201: Obtain the first audio recording of the user singing the target song and the second audio recording of the singer singing the target song.
[0032] The user's initial audio recording can be sent from the terminal to the server. In one example, the user can trigger an AI-generated audio recording function within a music application running on the terminal, thus entering the AI-generated audio recording interface. In this interface, the user can select a target song and trigger recording or uploading of their own recording of that song, thereby obtaining the initial audio recording. After the user triggers the audio recording generation in the AI-generated audio recording interface, the terminal, in response to the user's action, can send the song identifier corresponding to the target song and the user's initial audio recording to the server.
[0033] After receiving the first audio recording and the song identifier of the target song from the terminal, the server can store the first audio recording and then search for the corresponding second audio recording in the music library based on the song identifier. The music library stores audio recordings of songs performed by professional singers.
[0034] Step 202: Extract the semantic features and timbre features corresponding to the first singing audio, and extract the melody features corresponding to the second singing audio.
[0035] After the server obtains the first and second vocal audio files, it can use methods such as... Figure 1 The timbre extractor extracts the timbre features of the first singing audio, the semantic extractor extracts the semantic features corresponding to the first singing audio, and the melody extractor extracts the melody features corresponding to the second singing audio.
[0036] In one example, the timbre extractor can be implemented using algorithms such as ECAPA-TDNN, d-vector, and MFCC, while the melody extractor can be implemented using algorithms such as CREPE, Parselmouth, and WORLD. The semantic extractor can be implemented using algorithms such as HuberT and Whisper.
[0037] It is understandable that the first audio recording is actually composed of a sequence of articulatory units. The division of articulatory units can be pre-defined by technicians. For example, a word can be considered a single articulatory unit, a syllable a single articulatory unit, or a phoneme a single articulatory unit. Each articulatory unit can correspond to one or more audio frames, and the semantic features include semantic sub-features corresponding to each articulatory unit. The semantic sub-features corresponding to a single articulatory unit can be represented by a single feature vector, or by feature vectors extracted separately for each of the one or more audio frames corresponding to that articulatory unit.
[0038] Step 203: After aligning the first and second singing audios at the pronunciation unit level, determine the first time information corresponding to each pronunciation unit in the first singing audio and the second time information corresponding to each pronunciation unit in the second singing audio.
[0039] Since a singer's melodic features are matched with the singer's semantic features, such as pronunciation and accent, meaning the singer's melodic variations are influenced by the duration of each vocal unit, in this embodiment, to better match the singer's melodic features with the user's semantic features, the first and second singing audios can be aligned at the vocal unit level. It is understood that the first and second singing audios are recordings of the same song, so the first vocal unit sequence included in the first singing audio and the second vocal unit sequence included in the second singing audio are the same or substantially the same. Aligning the first and second singing audios at the vocal unit level means determining the correspondence between each vocal unit that is identical in timing within the first and second vocal unit sequences. That is, associating multiple vocal units included in the first vocal unit sequence with multiple vocal units included in the second vocal unit sequence, where the associated vocal units correspond to the same timing.
[0040] In the first singing audio, the first time information corresponding to each vocal unit can be the start and end time points of each vocal unit's pronunciation, or the first pronunciation duration corresponding to each vocal unit. In the second singing audio, the second time information corresponding to each vocal unit can be the start and end time points of each vocal unit's pronunciation, or the second pronunciation duration corresponding to each vocal unit.
[0041] Step 204: Determine the timing scaling parameters corresponding to each vocal unit based on the first and second time information.
[0042] After determining the first and second pronunciation information corresponding to each pronunciation unit, the ratio of the first pronunciation duration corresponding to the first pronunciation unit sequence to the second pronunciation duration corresponding to the second pronunciation unit sequence can be determined, thus determining the timing scaling parameter for each pronunciation unit. Here, each pronunciation unit can be a pronunciation unit in the second pronunciation unit sequence.
[0043] Step 205: Based on the timing scaling parameters corresponding to each articulation unit, adjust the timing of the melodic features corresponding to each articulation unit to obtain the target melodic features.
[0044] In one example, the melody features corresponding to the vocal unit include the melody sub-features corresponding to a first number of audio frames in the second singing audio. The scaling processing for the melody features includes: For each vocal unit, the product of the first number of audio frames corresponding to the vocal unit in the second singing audio and the corresponding temporal scaling parameter is determined to obtain the second number. The first number of melodic sub-features corresponding to the vocal unit are interpolated or pruned to obtain the second number of melodic sub-features, wherein the second number of melodic sub-features are the scaled melodic features.
[0045] In implementation, the melodic sub-features corresponding to each audio frame can be represented by a vector. After determining the temporal scaling parameter corresponding to a vocal unit, this parameter can be multiplied by the first number of audio frames corresponding to that vocal unit in the second singing audio to obtain the second number of melodic sub-features after scaling the melodic features of that vocal unit. If the first number is greater than the second number, the first number of melodic sub-features corresponding to the vocal unit can be pruned to obtain the second number of melodic sub-features, thus obtaining the scaled melodic features corresponding to that vocal unit. If the first number is less than the second number, the first number of melodic sub-features corresponding to the vocal unit can be interpolated to obtain the second number of melodic sub-features, thus obtaining the scaled melodic features corresponding to that vocal unit, which is the target melodic feature.
[0046] In one feasible approach, the first and second pronunciation durations corresponding to each articulation unit include an initial phase, a sustained phase, and an ending phase, respectively. For example, when the articulation unit is a word or syllable, each articulation unit may correspond to an initial phase, a sustained phase, and an ending phase. The identification of the initial phase, sustained phase, and ending phase can also be obtained during the extraction of semantic features.
[0047] The ratio of the first and second pronunciation durations for each pronunciation unit calculated above can respectively include a first ratio for the starting period, a second ratio for the duration, and a third ratio for the ending period corresponding to that pronunciation unit. Correspondingly, scaling the melody features according to these ratios can be performed by scaling the melody features of one or more audio frames corresponding to the starting period according to the first ratio, scaling the melody features of one or more audio frames corresponding to the duration according to the second ratio, and scaling the melody features of one or more audio frames corresponding to the ending period according to the third ratio. The scaling processing performed on the first, second, or third ratio can refer to the scaling processing performed based on the ratios described above, and will not be repeated here.
[0048] Figure 3 This is a schematic diagram of a melody corrector provided in an embodiment of this application. Figure 3 The melody corrector shown can perform scaling processing on melody features. For example... Figure 3 As shown, a phoneme- or word-level alignment model can be trained to force alignment of the phoneme sequences and their durations in the first and second audio recordings. Forced alignment refers to knowing the complete text content corresponding to the audio (e.g., phoneme sequences; if the phoneme is a word, the sequence can be root text). Instead of "guessing" the text through a speech recognition model, the model directly uses text to constrain the temporal segmentation of the audio, ultimately outputting a one-to-one correspondence between text units (phonemes / words) and audio segments. The input to the alignment model includes the phoneme sequence and the corresponding first or second audio recording. The output is the duration of each phoneme in the phoneme sequence, such as the start and end timestamps of the corresponding pronunciation period. The alignment model can be implemented using MFA (Montreal Forced Aligner) technology. For example, the model can be trained on a large amount of annotated text to accurately output the start and end timestamps corresponding to the duration of each phoneme in the first and second phoneme sequence pairs.
[0049] Step 206: Generate a third recording of the user singing the target song based on the target melody features, semantic features, and timbre features.
[0050] After scaling the melodic features of each vocal unit of the singer, the melodic features of each vocal unit of the singer will match the semantic features of each vocal unit of the user better. By inputting the target melodic features corresponding to each vocal unit of the singer, the semantic features of the user, and the timbre features into the acoustic model, the acoustic model can generate higher quality third-person singing audio.
[0051] Understandably, scaling the training features for each syllable unit is to match the user's pronunciation characteristics of each syllable unit. This is a detailed adjustment of the melody features and has no impact on the accuracy of the overall melody of the third-person audio of the generated target song.
[0052] In this embodiment, considering the possibility of a user's voice sounding unnatural or "plain," a method for enhancing the user's voice is also provided. This method involves identifying a reference timbre feature with the highest similarity to a timbre feature in a timbre library, wherein the timbre library includes timbre features corresponding to multiple singers. Based on the semantic sub-features of each vocal unit, the scaled melody features, and the reference timbre feature, a third audio recording of the user singing a target song is generated.
[0053] The server contains a professional singers' voice library. This library can store a large number of voice features corresponding to various singers. For example, the library could include voice features for singers of all ages and genders, and the number of singers included in the library would exceed a set threshold, such as 10,000. This ensures that the voice features matched to the user are close enough to be identified from the library. Each singer in the music library can correspond to one or more voice features, which can be extracted from one or more songs performed by the corresponding singer. The method for extracting reference voice features from the singer's song audio can be the same as the method for extracting voice features.
[0054] In implementation, after determining the user's timbre characteristics, a reference timbre feature with the highest similarity to the user's timbre can be calculated from the large number of timbre features included in the timbre library. Thus, the determined reference timbre feature is the singer's timbre that is closest to the user's timbre. Since singers' pronunciation is relatively professional, the timbre corresponding to the reference timbre feature is the timbre that is closest to the user's timbre and conforms to professional singing pronunciation. In one example, both the timbre features and the timbre features stored in the timbre library can be represented in vector form. The server calculates the similarity between the timbre feature and each timbre feature in the timbre library, which can be achieved by calculating the cosine similarity between the timbre feature and each other.
[0055] After determining the reference timbre feature that best matches the user's timbre characteristics, a third vocal audio recording of the user singing the target song can be generated based on the semantic sub-features of each articulation unit, the scaled melody features, and the reference timbre features. The process of generating the third vocal audio recording of the user singing the target song based on the semantic sub-features of each articulation unit, the scaled melody features, and the reference timbre features can be referenced to the process described above, and will not be elaborated further here.
[0056] This application does not directly input the user's timbre features into the acoustic model. Instead, it first queries the audio library for the singer's timbre that is closest to the user's, and then uses the corresponding reference timbre features of the singer's timbre as the reference timbre features for generating audio data in the acoustic model. In this way, the second singing audio generated for the user is not only close to the user's timbre, but also solves the problem of the user's "plain voice," thereby improving the quality of the singing audio generated for the user.
[0057] In this embodiment, the user's timbre features can be further fused with reference timbre features to obtain fused timbre features. This fused timbre feature is closer to the original timbre features and retains the timbre features of the singer's professional vocal style. Then, based on the generated fused timbre features, melody features, and semantic features, a second singing audio of the user performing the target song can be generated. Thus, the timbre of the generated second singing audio is even closer to the user's timbre, which also solves the problem of the user's "plain voice" timbre, further improving the quality of the singing audio generated for the user.
[0058] This application also provides a method for fusing timbre features and reference timbre features. The corresponding processing includes: weighting and summing the timbre features and reference timbre features according to a first weighting coefficient and a second weighting coefficient to obtain a fourth timbre feature; obtaining an average timbre feature, which is generated by the diffusion model under unrestricted input conditions; and enhancing the fourth timbre feature according to the average timbre feature to obtain a fused timbre feature.
[0059] In implementation, the aforementioned acoustic model can be a diffusion transformer (DIT) model. The average timbre feature can be the audio feature output by the diffusion model under unrestricted input conditions. It can be understood that the average timbre feature is a timbre feature without obvious timbre characteristics. Enhancing the fourth timbre feature based on the average timbre feature can be achieved by calculating the difference between the average timbre feature and the fourth timbre feature, and then using this difference to highlight the timbre characteristics corresponding to the fourth timbre feature. That is, highlighting the timbre characteristics corresponding to the weighted sum of the timbre feature and the reference timbre feature.
[0060] In this embodiment, the traditional Classifier-Free Guidance (CFG) technique corresponding to the diffusion model can be improved to achieve the fusion of timbre features and reference timbre features. The corresponding processing includes: obtaining the enhancement coefficient, determining the difference between the fourth timbre feature and the average timbre feature, determining the product of the difference and the enhancement coefficient, and summing the fourth timbre feature and the product to obtain the fused timbre feature.
[0061] The enhancement coefficients, specifically the first and second weighting coefficients used to calculate the fourth timbre feature, can both be preset by a technician. Traditional CFG technology can only enhance a single timbre feature; however, in this embodiment, it can achieve fusion enhancement of two timbre features. The following formula is an example of the improved CFG technology provided in this application.
[0062] z_gen = (1 + inference_cfg_rate) w1 z_A+(1.0 + inference_cfg_rate) w2 z_B - inference_cfg_rate z_uncond.
[0063] Where z_gen is the fused timbre feature obtained after fusion enhancement processing, z_A and z_B are the timbre feature and reference timbre feature respectively, inference_cfg_rate is the enhancement coefficient, w1 and w2 are the first weight coefficient and the second weight coefficient respectively, and z_uncond is the average timbre feature.
[0064] After obtaining the fused timbre features obtained by timbre fusion processing of the timbre features and reference timbre features, a third vocal audio of the user singing the target song is generated based on the semantic sub-features of each articulatory unit, the scaled melody features, and the fused timbre features. The process of generating the third vocal audio of the user singing the target song based on the semantic sub-features of each articulatory unit, the scaled melody features, and the fused timbre features can be referred to the process described above for generating the third vocal audio of the user singing the target song based on the semantic sub-features of each articulatory unit, the scaled melody features, and the timbre features; it will not be elaborated further here.
[0065] All of the above-mentioned optional technical solutions can be combined in any way to form optional embodiments of this disclosure, and will not be described in detail here.
[0066] Based on the same inventive concept, this application also provides a spatial audio file processing device, which can be configured on the server or terminal in the above embodiments. Figure 4 This is a schematic diagram of a device structure for generating singing audio provided in an embodiment of this application, as shown below. Figure 4 As shown, the device includes: The acquisition module 410 is used to acquire the first audio recording of the user singing the target song and the second audio recording of the singer singing the target song. Extraction module 420 is used to extract semantic features and timbre features corresponding to the first singing audio, and to extract melody features corresponding to the second singing audio; The determining module 430 is used to determine, after performing articulation unit-level alignment on the first singing audio and the second singing audio, the first timing information corresponding to each articulation unit in the first singing audio and the second timing information corresponding to each articulation unit in the second singing audio; and to determine the timing scaling parameter corresponding to each articulation unit based on the first timing information and the second timing information. The adjustment module 440 is used to perform time-series adjustment on the melodic features corresponding to each vocal unit according to the time-series scaling parameters corresponding to each vocal unit, so as to obtain the target melodic features; The generation module 450 is used to generate a third singing audio of the user singing the target song based on the target melody features, the semantic features, and the timbre features.
[0067] Optionally, the determining module 430 is configured to: Based on the first time information and the second time information, determine the first pronunciation duration of each pronunciation unit in the first singing audio and the second pronunciation duration in the second singing audio; The ratio of the first pronunciation duration and the second pronunciation duration corresponding to each pronunciation unit is determined as the timing scaling parameter for each pronunciation unit.
[0068] Optionally, the first and second pronunciation durations corresponding to each pronunciation unit respectively include the pronunciation durations corresponding to the start period, the duration, and the end period. The ratio of the first and second pronunciation durations includes the first ratio corresponding to the start period, the second ratio corresponding to the duration, and the third ratio corresponding to the end period. The adjustment module 440 is configured to: for each vocal unit, scale the melody features corresponding to the starting time period according to the first ratio, scale the melody features corresponding to the duration period according to the second ratio, and scale the melody features corresponding to the ending time period according to the third ratio.
[0069] Optionally, the melody features corresponding to the pronunciation unit include the melody sub-features corresponding to a first number of audio frames in the second singing audio; The adjustment module 440 is configured to: for each vocal unit, determine the product of the first number of audio frames corresponding to the vocal unit in the second singing audio and the ratio to obtain a second number; for each vocal unit, perform interpolation or deletion processing on the first number of melodic sub-features corresponding to the vocal unit to obtain the target melody feature composed of the second number of melody sub-features.
[0070] Optionally, the pronunciation unit includes phonemes, syllables, or words.
[0071] Optionally, the generation module 450 is used to determine a reference timbre feature that matches the timbre feature in a timbre library, wherein the timbre library includes timbre features corresponding to multiple singers; and to generate a third singing audio of the user singing the target song based on the target melody feature, the semantic feature and the reference timbre feature.
[0072] Optionally, the generation module 450 is used to perform timbre fusion processing on the timbre features and the matched reference timbre features to obtain fused timbre features; and to generate a third singing audio of the user singing the target song based on the target melody features, the semantic features and the fused timbre features.
[0073] It should be noted that the apparatus for generating singing audio provided in the above embodiments is only illustrated by the division of the above functional modules. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the server can be divided into different functional modules to complete all or part of the functions described above. In addition, the apparatus for generating singing audio and the method for generating singing audio provided in the above embodiments belong to the same concept, and the specific implementation process can be found in the method embodiments, which will not be repeated here.
[0074] Figure 5 This is a schematic diagram of the structure of a computer device provided in an embodiment of this application. The computer device 500 can be a server as described in the above embodiments. The computer device 500 can vary significantly due to different configurations or performance. It may include one or more central processing units (CPUs) 501 and one or more memories 502. The memory 502 stores at least one instruction, which is loaded and executed by the processor 501 to implement the methods provided in the various method embodiments described above. Of course, the server may also have wired or wireless network interfaces, a keyboard, and input / output interfaces for input and output. The server may also include other components for implementing device functions, which will not be elaborated upon here.
[0075] Figure 6 This illustration shows a structural block diagram of a computer device 600 provided in an exemplary embodiment of this application. The computer device 600 can be a portable mobile terminal, such as the terminal shown in the above embodiments, including smartphones, tablets, MP3 players (Moving Picture Experts Group Audio Layer III), MP4 players (Moving Picture Experts Group Audio Layer IV), laptops, or desktop computers. The computer device 600 may also be referred to as user equipment, portable terminal, laptop terminal, desktop terminal, or other names.
[0076] Typically, computer device 600 includes a processor 601 and a memory 602.
[0077] Processor 601 may include one or more processing cores, such as a quad-core processor, an octa-core processor, etc. Processor 601 may be implemented using at least one hardware form selected from DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), and PLA (Programmable Logic Array). Processor 601 may also include a main processor and a coprocessor. The main processor, also known as a CPU (Central Processing Unit), is used to process data in the wake-up state; the coprocessor is a low-power processor used to process data in the standby state. In some embodiments, processor 601 may integrate a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content required to be displayed on the screen. In some embodiments, processor 601 may also include an AI (Artificial Intelligence) processor, which is used to handle computational operations related to machine learning.
[0078] Memory 602 may include one or more computer-readable storage media, which may be non-transitory. Memory 602 may also include high-speed random access memory and non-volatile memory, such as one or more disk storage devices or flash memory devices. In some embodiments, the non-transitory computer-readable storage media in memory 602 is used to store at least one instruction, which is executed by processor 601 to implement the method for generating singing audio provided in the method embodiments of this application.
[0079] In some embodiments, the computer device 600 may optionally include a peripheral device interface 603 and at least one peripheral device. The processor 601, memory 602, and peripheral device interface 603 can be connected via a bus or signal line. Each peripheral device can be connected to the peripheral device interface 603 via a bus, signal line, or circuit board. Specifically, the peripheral device includes at least one of the following: a radio frequency circuit 604, a display screen 605, a camera assembly 606, an audio circuit 607, a positioning assembly 608, and a power supply 609.
[0080] Peripheral interface 603 can be used to connect at least one I / O (Input / Output) related peripheral device to processor 601 and memory 602. In some embodiments, processor 601, memory 602 and peripheral interface 603 are integrated on the same chip or circuit board; in some other embodiments, any one or two of processor 601, memory 602 and peripheral interface 603 can be implemented on separate chips or circuit boards, which is not limited in this embodiment.
[0081] The radio frequency (RF) circuit 604 is used to receive and transmit RF (Radio Frequency) signals, also known as electromagnetic signals. The RF circuit 604 communicates with communication networks and other communication devices via electromagnetic signals. The RF circuit 604 converts electrical signals into electromagnetic signals for transmission, or converts received electromagnetic signals back into electrical signals. Optionally, the RF circuit 604 includes: an antenna system, an RF transceiver, one or more amplifiers, a tuner, an oscillator, a digital signal processor, a codec chipset, a user identity module card, etc. The RF circuit 604 can communicate with other terminals through at least one wireless communication protocol. This wireless communication protocol includes, but is not limited to: the World Wide Web, metropolitan area networks, intranets, various generations of mobile communication networks (2G, 3G, 4G, and 5G), and wireless local area networks. In some embodiments, the RF circuit 604 may also include circuitry related to NFC (Near Field Communication), which is not limited in this application.
[0082] Display screen 605 is used to display a UI (User Interface). This UI may include graphics, text, icons, videos, and any combination thereof. When display screen 605 is a touch display screen, it also has the ability to collect touch signals on or above its surface. These touch signals can be input as control signals to processor 601 for processing. In this case, display screen 605 can also be used to provide virtual buttons and / or a virtual keyboard, also known as soft buttons and / or a soft keyboard. In some embodiments, display screen 605 may be a single screen, disposed on the front panel of computer device 600; in other embodiments, display screen 605 may be at least two screens, disposed on different surfaces of computer device 600 or in a folded design; in still other embodiments, display screen 605 may be a flexible display screen, disposed on a curved or folded surface of computer device 600. Furthermore, display screen 605 may be configured as a non-rectangular, irregular shape, i.e., a non-rectangular screen. Display screen 605 may be made of materials such as LCD (Liquid Crystal Display) or OLED (Organic Light-Emitting Diode).
[0083] The camera assembly 606 is used to acquire images or videos. Optionally, the camera assembly 606 includes a front-facing camera and a rear-facing camera. Typically, the front-facing camera is located on the front panel of the terminal, and the rear-facing camera is located on the back of the terminal. In some embodiments, there are at least two rear-facing cameras, which are any one of a main camera, a depth-sensing camera, a wide-angle camera, and a telephoto camera, to achieve background blurring by fusion of the main camera and the depth-sensing camera, panoramic shooting by fusion of the main camera and the wide-angle camera, VR (Virtual Reality) shooting, or other fusion shooting functions. In some embodiments, the camera assembly 606 may also include a flash. The flash can be a single-color temperature flash or a dual-color temperature flash. A dual-color temperature flash refers to a combination of a warm-light flash and a cool-light flash, which can be used for light compensation at different color temperatures.
[0084] The audio circuit 607 may include a microphone and a speaker. The microphone is used to collect sound waves from the user and the environment, converting them into electrical signals that are input to the processor 601 for processing, or to the radio frequency circuit 604 for voice communication. For stereo sound acquisition or noise reduction purposes, multiple microphones may be used, each positioned at a different location within the computer device 600. The microphone may also be an array microphone or an omnidirectional microphone. The speaker is used to convert electrical signals from the processor 601 or the radio frequency circuit 604 into sound waves. The speaker may be a conventional diaphragm speaker or a piezoelectric ceramic speaker. When the speaker is a piezoelectric ceramic speaker, it can convert electrical signals not only into audible sound waves but also into inaudible sound waves for purposes such as distance measurement. In some embodiments, the audio circuit 607 may also include a headphone jack.
[0085] The positioning component 608 is used to locate the current geographic location of the computer device 600 in order to enable navigation or LBS (Location Based Service).
[0086] Power supply 609 is used to supply power to the various components in computer device 600. Power supply 609 can be alternating current, direct current, a disposable battery, or a rechargeable battery. When power supply 609 includes a rechargeable battery, the rechargeable battery can be a wired rechargeable battery or a wireless rechargeable battery. A wired rechargeable battery is a battery that is charged via a wired line, while a wireless rechargeable battery is a battery that is charged via a wireless coil. The rechargeable battery can also be used to support fast charging technology.
[0087] Those skilled in the art will understand that Figure 6 The structure shown does not constitute a limitation on computer device 600, and may include more or fewer components than shown, or combine certain components, or use different component arrangements.
[0088] In an exemplary embodiment, a computer-readable storage medium is also provided, such as a memory including instructions that can be executed by a processor in a terminal to complete the method for generating vocal audio in the above embodiments. This computer-readable storage medium can be non-transitory. For example, the computer-readable storage medium can be ROM (Read-Only Memory), RAM (Random Access Memory), CD-ROM (CompactDisc Read-Only Memory), magnetic tape, floppy disk, and optical data storage devices, etc.
[0089] In an exemplary embodiment, a computer program product is also provided, which includes program code that can be executed by a processor in a computer device to perform the method for generating singing audio described above.
[0090] It should be noted that all information (including but not limited to user device information, user personal information, etc.), data (including but not limited to data used for analysis, stored data, displayed data, etc.), and signals (including but not limited to signals transmitted between user terminals and other devices) involved in this application have been authorized by the user or fully authorized by all parties, and the collection, use, and processing of related data must comply with the relevant laws, regulations, and standards of the relevant countries and regions. For example, the page information involved in this application was obtained with full authorization.
[0091] Those skilled in the art will understand that all or part of the steps of the above embodiments can be implemented by hardware or by a program instructing related hardware. The program can be stored in a computer-readable storage medium, such as a read-only memory, a disk, or an optical disk.
[0092] It should be understood that "multiple" as used herein refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. Furthermore, the step numbers described herein are merely illustrative of one possible execution order. In some other embodiments, the steps may not be executed in numerical order, such as two steps with different numbers being executed simultaneously, or two steps with different numbers being executed in the reverse order of the illustration. This application does not limit this.
[0093] The above description is merely an optional embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.
Claims
1. A method for generating singing audio, characterized in that, The method includes: Obtain the first audio recording of the user singing the target song and the second audio recording of the singer singing the target song; Extract the semantic features and timbre features corresponding to the first singing audio, and extract the melody features corresponding to the second singing audio; After aligning the first and second singing audios at the pronunciation unit level, the first time information corresponding to each pronunciation unit in the first singing audio and the second time information corresponding to each pronunciation unit in the second singing audio are determined. Based on the first time information and the second time information, determine the timing scaling parameters corresponding to each sound unit; Based on the timing scaling parameters corresponding to each vocal unit, the melodic features corresponding to each vocal unit are adjusted in timing to obtain the target melodic features; Based on the target melody features, the semantic features, and the timbre features, a third singing audio of the user singing the target song is generated.
2. The method according to claim 1, characterized in that, The step of determining the timing scaling parameters corresponding to each vocal unit based on the first timing information and the second timing information includes: Based on the first time information and the second time information, determine the first pronunciation duration of each pronunciation unit in the first singing audio and the second pronunciation duration in the second singing audio; The ratio of the first pronunciation duration and the second pronunciation duration corresponding to each pronunciation unit is determined as the timing scaling parameter for each pronunciation unit.
3. The method according to claim 2, characterized in that, The first and second pronunciation durations corresponding to each pronunciation unit respectively include the pronunciation durations corresponding to the start period, the duration, and the end period. The ratio of the first pronunciation duration to the second pronunciation duration includes the first ratio corresponding to the start period, the second ratio corresponding to the duration, and the third ratio corresponding to the end period. The step of adjusting the temporal features corresponding to each vocal unit according to the temporal scaling parameters includes: For each vocal unit, the melody features corresponding to the starting time period are scaled according to the first ratio, the melody features corresponding to the duration period are scaled according to the second ratio, and the melody features corresponding to the ending time period are scaled according to the third ratio.
4. The method according to claim 2, characterized in that, The melody features corresponding to the vocal unit include the melody sub-features corresponding to the first number of audio frames in the second singing audio; The step of adjusting the temporal features corresponding to each articulation unit according to the temporal scaling parameters to obtain the target melody features includes: For each vocal unit, the product of the first number of audio frames corresponding to the vocal unit in the second singing audio and the ratio is determined to obtain the second number; For each vocal unit, the first number of melodic sub-features corresponding to the vocal unit are interpolated or pruned to obtain the target melodic feature composed of the second number of melodic sub-features.
5. The method according to claim 1, characterized in that, The pronunciation unit includes phonemes, syllables, or words.
6. The method according to any one of claims 1 to 5, characterized in that, The step of generating a third vocal audio recording of the user singing the target song based on the target melody features, the semantic features, and the timbre features includes: In the timbre library, a reference timbre feature matching the timbre feature is determined, wherein the timbre library includes timbre features corresponding to multiple singers; Based on the target melody features, the semantic features, and the reference timbre features, a third singing audio of the user singing the target song is generated.
7. The method according to claim 6, characterized in that, The step of generating a third vocal audio recording of the user singing the target song based on the target melody features, the semantic features, and the reference timbre features includes: The timbre features are fused with the matched reference timbre features to obtain fused timbre features; Based on the target melody features, the semantic features, and the fused timbre features, a third vocal audio recording of the user singing the target song is generated.
8. A computer device, characterized in that, The computer device includes a processor and a memory, the memory storing at least one instruction that is loaded and executed by the processor to perform the operations performed by the method for generating vocal audio as described in any one of claims 1 to 7.
9. A computer-readable storage medium, characterized in that, The storage medium stores at least one instruction, which is loaded and executed by a processor to perform the operations performed by the method for generating vocal audio as described in any one of claims 1 to 7.
10. A computer program product, characterized in that, The storage medium stores at least one piece of program code, which is loaded and executed by a processor to perform the operations of the method for generating vocal audio as described in any one of claims 1 to 8.