Method, computer device, storage medium and program product for generating a sung audio

Abstract

The embodiment disclosed by the application belongs to the technical field of artificial intelligence, and particularly relates to a method for generating singing audio, a computer device, a storage medium and a program product. The method comprises the following steps: obtaining a recording audio of a user and a first singing audio of a singer singing a target song; extracting a first timbre feature corresponding to the recording audio and a singing feature corresponding to the first singing audio; determining a second timbre feature matched with the first timbre feature in a timbre library, wherein the timbre library comprises timbre features corresponding to multiple singers; fusing the first timbre feature and the second timbre feature to obtain a fused timbre feature; and generating a second singing audio of the user singing the target song according to the fused timbre feature and the singing feature. By using the application, the quality of the generated singing audio for the user can be improved.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a method, computer device, storage medium, and program product for generating vocal audio. Background Technology

[0002] Singing Voice Conversion (SVC) is an AI-powered voice conversion technology that can transform one person's singing voice into another's. For example, it can convert the voice of a professional singer into the voice of an ordinary user, thereby increasing the flexibility and enjoyment of music playback.

[0003] Current vocal conversion technology uses artificial intelligence to extract a user's timbre from their recorded audio and then replaces it with the timbre from a professional singer's recording. However, since users generally haven't received professional vocal training, their timbre in the recorded audio is usually their speaking timbre—a typical "plain voice." Therefore, the quality of the vocal audio generated based on the user's timbre is relatively low. Summary of the Invention

[0004] This application provides a method, computer device, storage medium, and program product for generating vocal audio, which can improve the quality of vocal audio generated for users. The corresponding technical solutions are as follows: In a first aspect, a method for generating vocal audio is provided. The method includes: acquiring a user's recorded audio and a first vocal audio of a singer performing a target song; extracting a first timbre feature corresponding to the recorded audio and a vocal feature corresponding to the first vocal audio; determining a second timbre feature matching the first timbre feature in a timbre library, wherein the timbre library includes timbre features corresponding to multiple singers; fusing the first timbre feature and the second timbre feature to obtain a fused timbre feature; and generating a second vocal audio of the user performing the target song based on the fused timbre feature and the vocal feature.

[0005] Optionally, the singing features include the melody features and semantic features corresponding to the first singing audio.

[0006] Optionally, fusing the first timbre feature and the second timbre feature to obtain the fused timbre feature includes: obtaining a first weighting coefficient corresponding to the first timbre feature and a second weighting coefficient corresponding to the second timbre feature; and performing weighted processing on the first timbre feature and the second timbre feature according to the first weighting coefficient and the second weighting coefficient to obtain the fused timbre feature.

[0007] Optionally, the step of weighting the first timbre feature and the second timbre feature according to the first weighting coefficient and the second weighting coefficient to obtain the fused timbre feature includes: weighting and summing the first timbre feature and the second timbre feature according to the first weighting coefficient and the second weighting coefficient to obtain the third timbre feature; obtaining the average timbre feature, wherein the average timbre feature is generated by the diffusion model under unrestricted input conditions, wherein the diffusion model is used to generate the second singing audio according to the fused timbre feature, the melody feature and the semantic feature; and enhancing the third timbre feature according to the average timbre feature to obtain the fused timbre feature.

[0008] Optionally, the step of enhancing the third timbre feature based on the average timbre feature to obtain the fused timbre feature includes: obtaining an enhancement coefficient; determining the difference between the third timbre feature and the average timbre feature, determining the product of the difference and the enhancement coefficient; and summing the third timbre feature and the product to obtain the fused timbre feature.

[0009] Optionally, generating the second singing audio of the user singing the target song based on the fused timbre features and the singing features includes: inputting the fused timbre features, the melody features, and the semantic features into a diffusion model, so that the diffusion model generates audio data matching the fused timbre features based on the melody features and the semantic features; and inputting the audio data into a vocoder to obtain the second singing audio output by the vocoder.

[0010] Optionally, the recorded audio is a third recording of a user singing the target song. The step of extracting the first timbre feature corresponding to the recorded audio includes: acquiring multiple audio frames included in a specified singing period in the third recording; extracting the fourth timbre feature corresponding to each of the multiple audio frames; and performing convergence processing on the fourth timbre features corresponding to the multiple audio frames to obtain the first timbre feature corresponding to the recorded audio.

[0011] Secondly, an apparatus for generating vocal audio is provided, the apparatus comprising: The acquisition module is used to acquire the user's recorded audio and the first audio recording of the singer performing the target song; The extraction module is used to extract the first timbre feature corresponding to the recorded audio and the singing feature corresponding to the first singing audio. A matching module is used to determine a second timbre feature that matches the first timbre feature in a timbre library, wherein the timbre library includes timbre features corresponding to multiple singers; The fusion module is used to fuse the first timbre feature with the second timbre feature to obtain a fused timbre feature; The generation module is used to generate a second singing audio of the user singing the target song based on the fused timbre features and the singing features.

[0012] Optionally, the singing features include the melody features and semantic features corresponding to the first singing audio.

[0013] Optionally, the fusion module is used to obtain a first weighting coefficient corresponding to the first timbre feature and a second weighting coefficient corresponding to the second timbre feature; and to perform weighted processing on the first timbre feature and the second timbre feature according to the first weighting coefficient and the second weighting coefficient to obtain the fused timbre feature.

[0014] Optionally, the fusion module is used to perform a weighted summation of the first timbre feature and the second timbre feature according to a first weight coefficient and a second weight coefficient to obtain a third timbre feature; obtain an average timbre feature, wherein the average timbre feature is generated by the diffusion model under unrestricted input conditions, wherein the diffusion model is used to generate the second singing audio according to the fused timbre feature, the melody feature and the semantic feature; and enhance the third timbre feature according to the average timbre feature to obtain the fused timbre feature.

[0015] Optionally, the fusion module is used to obtain an enhancement coefficient; determine the difference between the third timbre feature and the average timbre feature; determine the product of the difference and the enhancement coefficient; and sum the third timbre feature and the product to obtain the fused timbre feature.

[0016] Optionally, the matching module is configured to: input the fused timbre feature, the melody feature, and the semantic feature into a diffusion model, so that the diffusion model generates audio data matching the fused timbre feature based on the melody feature and the semantic feature; and input the audio data into a vocoder to obtain the second singing audio output by the vocoder.

[0017] Optionally, the recorded audio is a third recording of a user singing the target song, and the extraction module is used to: acquire multiple audio frames included in a specified singing period in the third recording; extract a fourth timbre feature corresponding to each of the multiple audio frames; and perform convergence processing on the fourth timbre features corresponding to the multiple audio frames to obtain a first timbre feature corresponding to the recorded audio.

[0018] Thirdly, a computer device is provided, the computer device including a processor and a memory, the memory storing at least one instruction, the instruction being loaded and executed by the processor to perform the operations performed by the method for generating vocal audio as described in the first aspect above.

[0019] Fourthly, a computer-readable storage medium is provided, the storage medium storing at least one instruction, the instruction being loaded and executed by a processor to perform the operations performed by the method for generating vocal audio as described in the first aspect above.

[0020] Fifthly, a computer program product is provided, wherein at least one piece of program code is stored in the storage medium, the program code being loaded and executed by a processor to perform the operations performed by the method for generating singing audio as described in the first aspect above.

[0021] The beneficial effects of the technical solution provided in this application are: In the technical solution provided in this application, a vocal timbre library corresponding to professional singers can be pre-set. When generating singing audio for a user, the most similar singer's timbre can be matched to the user in the vocal timbre library. Then, the singer's timbre and the user's timbre are fused, and the singing audio of the target song is generated for the user based on the fused timbre. In this way, the timbre of the generated singing audio is close to the user's timbre, and the influence of the user's "plain voice" on the quality of the generated singing audio can be reduced by utilizing the singer's timbre. Attached Figure Description

[0022] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0023] Figure 1 This is a schematic diagram of a traditional method for generating audio of a user singing a target song; Figure 2 This is a flowchart of a method for generating singing audio provided in an embodiment of this application; Figure 3 This is a schematic diagram of a method for generating singing audio provided in an embodiment of this application; Figure 4 This is a schematic diagram of a method for generating singing audio provided in an embodiment of this application; Figure 5 This is a schematic diagram of a device for generating singing audio provided in an embodiment of this application; Figure 6This is a schematic diagram of the structure of a computer device provided in an embodiment of this application; Figure 7 This is a schematic diagram of the structure of a computer device provided in an embodiment of this application. Detailed Implementation

[0024] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application will be described in further detail below with reference to the accompanying drawings.

[0025] Vocal synthesis technology is a speech synthesis technique designed to enable computers to produce beautiful and melodious singing voices, much like humans. As a new application area of ​​speech synthesis technology, vocal synthesis has considerable application value and promising prospects in fields such as virtual singers, record production, and digital music creation.

[0026] Vocal synthesis technology utilizes artificial intelligence to generate audio of the user singing a target song, based on the user's timbre. The generated audio sounds just like the user actually singing. Figure 1 This is a schematic diagram of a traditional method for generating audio of a user singing a target song. (Example) Figure 1 As shown, the process first acquires the introductory audio of the target song and the user's recorded audio. The target song can be the song for which the user wishes to generate a performance audio; the introductory audio can be the performance audio of the target song sung by a professional singer; and the user's recorded audio can be audio of the user speaking or singing a song for a certain duration, captured from the user's terminal.

[0027] In one example, the terminal's running music application has an AI-generated singing audio function. Users can record or upload their own audio (i.e., record a song) within the corresponding interface and select the song they want to sing (i.e., the target song). The terminal can then send the recorded audio and the target song's identifier to the server, which will then execute... Figure 1 The method shown is for generating audio of a user singing a target song.

[0028] Understandably, the lead vocal audio provides the melody and semantic features of the vocal audio to be generated, while the user's recorded audio provides the user's timbre features. See also... Figure 1The system can input the lead vocal audio into both the melody extractor and the semantic extractor, obtaining the melody features output by the melody extractor and the semantic features output by the semantic extractor based on the lead vocal audio. Similarly, the user's recorded audio can be input into the timbre extractor to obtain the user's timbre features. Then, the melody features, semantic features, and timbre features can be input into the acoustic model to obtain the audio data output by the acoustic model. Finally, the vocoder decodes the audio data to obtain a performance audio of the target song generated according to the user's timbre and the singing skills of a professional singer.

[0029] exist Figure 1 The method shown combines a professional singer's vocal technique with the user's timbre to generate audio of the user singing the target song. However, the user's recorded audio is typically either spoken or sung. Since users generally haven't received professional vocal training, both spoken and sung audio are typically produced with a "plain voice," not a professional singing style. Therefore, the timbre extracted from the user's recorded audio is their "plain voice," and the generated audio is also based on this "plain voice." It's understandable that the purpose of generating audio through vocal synthesis technology is to predict the user's performance after professional vocal training. However, the presence of the user's "plain voice" causes the generated audio to not match the performance expected after professional training, resulting in low-quality audio.

[0030] This application provides a method for generating singing audio. In this method, a vocal library corresponding to professional singers can be pre-set. When generating singing audio for a user, the most similar singer's vocal timbre can be matched to the user in the vocal library, and then the singing audio of the target song can be generated based on the singer's vocal timbre. In this way, the generated singing audio has a vocal timbre that is close to the user's own vocal timbre, while avoiding the degradation of quality caused by a "plain voice" vocal timbre.

[0031] Figure 2 This is a flowchart illustrating a method for generating vocal audio according to an embodiment of this application. This method can be executed by a server, such as... Figure 2 As shown, the method includes: Step 201: Obtain the user's recorded audio and the first recording of the singer performing the target song.

[0032] The user's recorded audio can be sent from the user's device to the server. In one example, a user can trigger an AI-generated singing audio function within a music application running on their device, thus entering the AI-generated singing audio interface. In this interface, the user can select a target song and trigger recording or uploading previously recorded audio to obtain their own recorded audio. This recorded audio can be either the user's spoken audio or their singing of the target song. After selecting a target song and recording or uploading the audio, the user can trigger the generation of the singing audio in the AI-generated singing audio interface. Responding to the user's trigger action, the device can send the song identifier corresponding to the target song and the user's audio to the server.

[0033] After receiving the recorded audio and the song identifier of the target song from the terminal, the server can store the recorded audio and search for the first recorded audio of the target song in the music library based on the song identifier. The music library stores recorded audio of songs sung by professional singers.

[0034] Step 202: Extract the first timbre feature corresponding to the recorded audio and the singing feature corresponding to the first singing audio.

[0035] In one example, performance features are used to characterize the singer's performance information when singing a target song, including at least one of melody features, semantic features, rhythmic features, prosodic features, temporal features, and emotional features. The following description uses an example where performance features include melody features and semantic features to illustrate an embodiment of this application.

[0036] After the server obtains the recorded audio and the first performance audio, it can use methods such as... Figure 1 The timbre extractor shown extracts the timbre features of the recorded audio to obtain the first timbre feature corresponding to the recorded audio. The melody extractor extracts the melody feature corresponding to the first singing audio, and the semantic extractor extracts the semantic feature corresponding to the first singing audio.

[0037] In one example, the timbre extractor can be implemented using algorithms such as ECAPA-TDNN, d-vector, and MFCC, while the melody extractor can be implemented using algorithms such as CREPE, Parselmouth, and WORLD. The semantic extractor can be implemented using algorithms such as HuberT and Whisper.

[0038] Step 203: Determine the second timbre feature that matches the first timbre feature in the timbre library, wherein the timbre library includes timbre features corresponding to multiple singers.

[0039] The server contains a professional singers' voice library. This library stores a large number of voice features corresponding to various singers. For example, the library includes voice features for singers of all ages and genders, and the number of singers included in the library exceeds a set threshold, such as 10,000. This ensures that the voice features matched to the user are close enough to match their vocal characteristics. Each singer in the music library can correspond to one or more voice features, which can be extracted from one or more songs performed by the corresponding singer. The method for extracting voice features from the singer's song audio can be the same as the method for extracting the first voice feature.

[0040] In implementation, after determining the user's first timbre feature, one or more second timbre features with the highest similarity to the first timbre feature can be calculated from the large number of timbre features included in the timbre library. Thus, the determined second timbre feature is the singer's timbre that is closest to the user's timbre, i.e., the second timbre feature that matches the first timbre feature. Since singers' pronunciation is relatively professional, the timbre corresponding to the second timbre feature is the timbre that is closest to the user's timbre and conforms to professional singing pronunciation. In one example, both the first timbre feature and the timbre features stored in the timbre library can be represented in vector form. The server calculates the similarity between the first timbre feature and each timbre feature in the timbre library, which can be achieved by calculating the cosine similarity between the first timbre feature and each timbre feature.

[0041] In one feasible approach, the recorded audio collected from the user can be a third recording of the user singing a target song. The first timbre feature extracted from the recorded audio can be the timbre features of multiple audio frames corresponding to a specified singing period in the third recording.

[0042] In practice, after selecting a target song in the AI-generated audio interface, users can begin recording their own rendition of the target song (the third audio recording). Users can sing part or all of the target song. The server, upon receiving the user's third audio recording, can extract audio frames corresponding to a specified time segment. This specified time segment can be the chorus portion of the target song sung by the user.

[0043] After obtaining multiple audio frames from a specified singing period in the third singing audio, the server can extract the fourth timbre features corresponding to each audio frame. These fourth timbre features are then aggregated to obtain the first timbre features corresponding to the recorded audio. This aggregation process can involve concatenating the fourth timbre features from multiple audio frames. By extracting the timbre features corresponding to a specified singing period of the user's target song, the server can obtain the true timbre features of the target song. This allows the server to match second timbre features from the timbre library that are closer to the user's singing target, thereby further improving the quality of the subsequently generated user-sung audio.

[0044] Step 204: Fuse the first timbre feature with the second timbre feature to obtain the fused timbre feature.

[0045] After obtaining the second timbre feature, the user's first timbre feature can be further fused with the second timbre feature to obtain a fused timbre feature. This fused timbre feature closely resembles the user's first timbre feature and retains the timbre characteristics of the singer's professional vocal technique. Then, based on the generated fused timbre feature, melody feature, and semantic feature, a second vocal audio recording of the user singing the target song can be generated. This results in a second vocal audio recording with a timbre even closer to the user's own timbre, effectively addressing the issue of the user's "plain voice" and further improving the quality of the generated vocal audio.

[0046] In one example, the fusion process of the first timbre feature and the second timbre feature may include calculating the mean of the first timbre feature and the second timbre feature. Alternatively, an technician may pre-set a first weighting coefficient corresponding to the first timbre feature and a second weighting coefficient corresponding to the second timbre feature in a server. The fusion process of the first weighting coefficient and the second weighting coefficient may include weighting the first timbre feature and the second timbre feature to obtain the fused timbre feature.

[0047] This application embodiment also provides a method for fusing a first timbre feature and a second timbre feature. The corresponding processing includes: weighting and summing the first timbre feature and the second timbre feature according to a first weighting coefficient and a second weighting coefficient to obtain a third timbre feature; obtaining an average timbre feature, which is generated by the diffusion model under unrestricted input conditions; and enhancing the third timbre feature according to the average timbre feature to obtain a fused timbre feature.

[0048] In implementation, the aforementioned acoustic model can be a diffusion transformer (DIT) model. The average timbre feature can be the audio feature output by the diffusion model under unrestricted input conditions. It can be understood that the average timbre feature is a timbre feature without obvious timbre characteristics. Enhancing the third timbre feature based on the average timbre feature can be achieved by calculating the difference between the average timbre feature and the third timbre feature, and then using this difference to highlight the timbre characteristics corresponding to the third timbre feature. That is, highlighting the timbre characteristics corresponding to the weighted sum of the first and second timbre features.

[0049] In this embodiment, the traditional Classifier-Free Guidance (CFG) technique corresponding to the diffusion model can be improved to achieve the fusion of the first timbre feature and the second timbre feature. The corresponding processing includes: obtaining an enhancement coefficient, determining the difference between the third timbre feature and the average timbre feature, determining the product of the difference and the enhancement coefficient, and summing the third timbre feature and the product to obtain the fused timbre feature.

[0050] The enhancement coefficients, specifically the first and second weighting coefficients used to calculate the third timbre feature, can both be preset by a technician. Traditional CFG technology can only enhance a single timbre feature; however, in this embodiment, it can achieve fusion enhancement of two timbre features. The following formula is an example of the improved CFG technology provided in this application.

[0051] z_gen = (1 + inference_cfg_rate) w1 z_A+(1.0 + inference_cfg_rate) w2 z_B - inference_cfg_rate z_uncond.

[0052] Where z_gen is the fused timbre feature obtained after fusion enhancement processing, z_A and z_B are the first timbre feature and the second timbre feature respectively, inference_cfg_rate is the enhancement coefficient, w1 and w2 are the first weight coefficient and the second weight coefficient respectively, and z_uncond is the average timbre feature.

[0053] Step 205: Based on the fused timbre characteristics and singing characteristics, generate a second singing audio of the user singing the target song.

[0054] Figure 3This is a schematic diagram illustrating a method for generating singing audio provided in an embodiment of this application. For example... Figure 3 As shown, after fusing and enhancing the first and second timbre features using the CFG technology provided in this application, the resulting fused timbre features, melodic features, and semantic features can be input into a diffusion model, so that the diffusion model generates audio data matching the fused timbre features based on the melodic and semantic features. The audio data can then be input into a vocoder to obtain the second vocal audio output by the vocoder.

[0055] Optionally, the fusion processing of the first and second timbre features can be triggered when the similarity between the first and second timbre features is less than a set similarity threshold. This fusion process allows the generated fused timbre features to produce a second vocal audio for the user. The resulting second vocal audio is closer to the user's timbre and possesses the singing skills of a professional singer. If the similarity between the first and second timbre features is less than the set similarity threshold, the second timbre features can be directly used to generate the second vocal audio for the user. This ensures the quality of the generated second vocal audio while reducing the computational load, thereby improving the efficiency of generating the second vocal audio. Directly using the second timbre features to generate the second vocal audio for the user can include generating the second vocal audio of the target song sung by the user based on the second timbre features, the melody features corresponding to the first vocal audio, and semantic features. For example, the second timbre features, melody features, and semantic features can be input into an acoustic model to obtain the audio data output by the acoustic model. Then, a vocoder decodes the audio data to obtain the second vocal audio generated for the user.

[0056] Figure 4 This is a schematic diagram illustrating a method for generating vocal audio provided in an embodiment of this application. Figure 1 Unlike the methods shown, this embodiment does not directly input the user's first timbre feature into the acoustic model. Instead, it first queries the audio library for the singer's timbre that is closest to the user's, and then uses the singer's timbre corresponding to the second timbre feature as the reference timbre feature for the acoustic model to generate audio data. In this way, the second singing audio generated for the user is not only close to the user's timbre, but also solves the problem of the user's "plain voice," thereby improving the quality of the singing audio generated for the user.

[0057] In this embodiment, the user's first timbre feature and second timbre feature can be further fused to obtain a fused timbre feature. This fused timbre feature is closer to the first timbre feature and retains the timbre features of the singer's professional vocal style. Then, based on the generated fused timbre feature, melody feature, and semantic feature, a second singing audio of the user performing the target song can be generated. This results in a second singing audio with a timbre closer to the user's own timbre, effectively addressing the issue of the user's "plain voice" and further improving the quality of the generated singing audio.

[0058] All of the above-mentioned optional technical solutions can be combined in any way to form optional embodiments of this disclosure, and will not be described in detail here.

[0059] Based on the same inventive concept, this application also provides an apparatus for generating singing audio, which can be configured on the server or terminal in the above embodiments. Figure 5 This is a schematic diagram of a device structure for generating singing audio provided in an embodiment of this application, as shown below. Figure 5 As shown, the device includes: The acquisition module 510 is used to acquire the user's recorded audio and the first recording audio of the singer performing the target song; Extraction module 520 is used to extract the first timbre feature corresponding to the recorded audio and the singing feature corresponding to the first singing audio; The matching module 530 is used to determine a second timbre feature that matches the first timbre feature in the timbre library, wherein the timbre library includes timbre features corresponding to multiple singers; The fusion module 540 is used to fuse the first timbre feature with the second timbre feature to obtain a fused timbre feature; The generation module 550 is used to generate a second singing audio of the user singing the target song based on the fused timbre features and the singing features.

[0060] Optionally, the singing features include the melody features and semantic features corresponding to the first singing audio.

[0061] Optionally, the fusion module 540 is used to obtain a first weighting coefficient corresponding to the first timbre feature and a second weighting coefficient corresponding to the second timbre feature; and to perform weighted processing on the first timbre feature and the second timbre feature according to the first weighting coefficient and the second weighting coefficient to obtain the fused timbre feature.

[0062] Optionally, the fusion module 540 is used to perform a weighted summation of the first timbre feature and the second timbre feature according to the first weight coefficient and the second weight coefficient to obtain a third timbre feature; obtain an average timbre feature, which is generated by the diffusion model under unrestricted input conditions, wherein the diffusion model is used to generate the second singing audio according to the fused timbre feature, the melody feature and the semantic feature; and enhance the third timbre feature according to the average timbre feature to obtain the fused timbre feature.

[0063] Optionally, the fusion module 540 is used to obtain an enhancement coefficient; determine the difference between the third timbre feature and the average timbre feature; determine the product of the difference and the enhancement coefficient; and sum the third timbre feature and the product to obtain the fused timbre feature.

[0064] Optionally, the matching module 530 is configured to: input the fused timbre feature, the melody feature, and the semantic feature into a diffusion model, so that the diffusion model generates audio data matching the fused timbre feature based on the melody feature and the semantic feature; and input the audio data into a vocoder to obtain the second singing audio output by the vocoder.

[0065] Optionally, the recorded audio is a third recording of a user singing the target song. The extraction module 520 is used to: acquire multiple audio frames included in a specified singing period in the third recording; extract the fourth timbre feature corresponding to each audio frame in the multiple audio frames; and perform convergence processing on the fourth timbre features corresponding to the multiple audio frames to obtain the first timbre feature corresponding to the recorded audio.

[0066] It should be noted that the apparatus for generating singing audio provided in the above embodiments is only illustrated by the division of the above functional modules. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the server can be divided into different functional modules to complete all or part of the functions described above. In addition, the apparatus for generating singing audio and the method for generating singing audio provided in the above embodiments belong to the same concept, and the specific implementation process can be found in the method embodiments, which will not be repeated here.

[0067] Figure 6This is a schematic diagram of the structure of a computer device provided in an embodiment of this application. The computer device can be the server described in the above embodiments. The computer device 600 can vary significantly due to different configurations or performance. It may include one or more central processing units (CPUs) 601 and one or more memories 602. The memory 602 stores at least one instruction, which is loaded and executed by the processor 601 to implement the methods provided in the various method embodiments described above. Of course, the server may also have wired or wireless network interfaces, a keyboard, and input / output interfaces for input and output. The server may also include other components for implementing device functions, which will not be elaborated here.

[0068] Figure 7 This illustration shows a structural block diagram of a computer device 700 provided in an exemplary embodiment of this application. The computer device 700 can be a portable mobile terminal, such as the terminal shown in the above embodiments, including smartphones, tablets, MP3 players (Moving Picture Experts Group Audio Layer III), MP4 players (Moving Picture Experts Group Audio Layer IV), laptops, or desktop computers. The computer device 700 may also be referred to as user equipment, portable terminal, laptop terminal, desktop terminal, or other names.

[0069] Typically, computer device 700 includes a processor 701 and a memory 702.

[0070] Processor 701 may include one or more processing cores, such as a quad-core processor, an octa-core processor, etc. Processor 701 may be implemented using at least one hardware form selected from DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), and PLA (Programmable Logic Array). Processor 701 may also include a main processor and a coprocessor. The main processor, also known as a CPU (Central Processing Unit), is used to process data in the wake-up state; the coprocessor is a low-power processor used to process data in the standby state. In some embodiments, processor 701 may integrate a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content required to be displayed on the screen. In some embodiments, processor 701 may also include an AI (Artificial Intelligence) processor, which is used to handle computational operations related to machine learning.

[0071] The memory 702 may include one or more computer-readable storage media, which may be non-transitory. The memory 702 may also include high-speed random access memory and non-volatile memory, such as one or more disk storage devices or flash memory devices. In some embodiments, the non-transitory computer-readable storage media in the memory 702 are used to store at least one instruction, which is executed by the processor 701 to implement the method for generating vocal audio provided in the embodiments of this application.

[0072] In some embodiments, the computer device 700 may optionally include a peripheral device interface 703 and at least one peripheral device. The processor 701, memory 702, and peripheral device interface 703 can be connected via a bus or signal line. Each peripheral device can be connected to the peripheral device interface 703 via a bus, signal line, or circuit board. Specifically, the peripheral device includes at least one of the following: a radio frequency circuit 704, a display screen 705, a camera assembly 706, an audio circuit 707, a positioning assembly 708, and a power supply 709.

[0073] Peripheral device interface 703 can be used to connect at least one I / O (Input / Output) related peripheral device to processor 701 and memory 702. In some embodiments, processor 701, memory 702 and peripheral device interface 703 are integrated on the same chip or circuit board; in some other embodiments, any one or two of processor 701, memory 702 and peripheral device interface 703 can be implemented on separate chips or circuit boards, which is not limited in this embodiment.

[0074] The radio frequency (RF) circuit 704 is used to receive and transmit RF (Radio Frequency) signals, also known as electromagnetic signals. The RF circuit 704 communicates with communication networks and other communication devices via electromagnetic signals. The RF circuit 704 converts electrical signals into electromagnetic signals for transmission, or converts received electromagnetic signals back into electrical signals. Optionally, the RF circuit 704 includes: an antenna system, an RF transceiver, one or more amplifiers, a tuner, an oscillator, a digital signal processor, a codec chipset, a user identity module card, etc. The RF circuit 704 can communicate with other terminals through at least one wireless communication protocol. This wireless communication protocol includes, but is not limited to: the World Wide Web, metropolitan area networks, intranets, various generations of mobile communication networks (2G, 3G, 4G, and 5G), and wireless local area networks. In some embodiments, the RF circuit 704 may also include circuitry related to NFC (Near Field Communication), which is not limited in this application.

[0075] Display screen 705 is used to display a UI (User Interface). This UI may include graphics, text, icons, videos, and any combination thereof. When display screen 705 is a touch display screen, it also has the ability to collect touch signals on or above its surface. These touch signals can be input as control signals to processor 701 for processing. In this case, display screen 705 can also be used to provide virtual buttons and / or a virtual keyboard, also known as soft buttons and / or a soft keyboard. In some embodiments, display screen 705 may be a single screen, disposed on the front panel of computer device 700; in other embodiments, display screen 705 may be at least two screens, disposed on different surfaces of computer device 700 or in a folded design; in still other embodiments, display screen 705 may be a flexible display screen, disposed on a curved or folded surface of computer device 700. Furthermore, display screen 705 may be configured as a non-rectangular, irregular shape, i.e., a non-rectangular screen. Display screen 705 may be made of materials such as LCD (Liquid Crystal Display) or OLED (Organic Light-Emitting Diode).

[0076] The camera assembly 706 is used to acquire images or videos. Optionally, the camera assembly 706 includes a front-facing camera and a rear-facing camera. Typically, the front-facing camera is located on the front panel of the terminal, and the rear-facing camera is located on the back of the terminal. In some embodiments, there are at least two rear-facing cameras, which are any one of a main camera, a depth-sensing camera, a wide-angle camera, and a telephoto camera, to achieve background blurring by fusion of the main camera and the depth-sensing camera, panoramic shooting by fusion of the main camera and the wide-angle camera, VR (Virtual Reality) shooting, or other fusion shooting functions. In some embodiments, the camera assembly 706 may also include a flash. The flash can be a single-color temperature flash or a dual-color temperature flash. A dual-color temperature flash refers to a combination of a warm-light flash and a cool-light flash, which can be used for light compensation at different color temperatures.

[0077] The audio circuit 707 may include a microphone and a speaker. The microphone is used to collect sound waves from the user and the environment, converting the sound waves into electrical signals that are input to the processor 701 for processing, or input to the radio frequency circuit 704 for voice communication. For stereo sound acquisition or noise reduction purposes, multiple microphones may be used, each located in a different part of the computer device 700. The microphone may also be an array microphone or an omnidirectional microphone. The speaker is used to convert electrical signals from the processor 701 or the radio frequency circuit 704 into sound waves. The speaker may be a conventional diaphragm speaker or a piezoelectric ceramic speaker. When the speaker is a piezoelectric ceramic speaker, it can convert electrical signals not only into audible sound waves but also into inaudible sound waves for purposes such as distance measurement. In some embodiments, the audio circuit 707 may also include a headphone jack.

[0078] The positioning component 708 is used to locate the current geographic location of the computer device 700 in order to enable navigation or LBS (Location Based Service).

[0079] Power supply 709 is used to supply power to the various components in computer device 700. Power supply 709 can be AC ​​power, DC power, a disposable battery, or a rechargeable battery. When power supply 709 includes a rechargeable battery, the rechargeable battery can be a wired rechargeable battery or a wireless rechargeable battery. A wired rechargeable battery is a battery that is charged via a wired line, while a wireless rechargeable battery is a battery that is charged via a wireless coil. The rechargeable battery can also be used to support fast charging technology.

[0080] Those skilled in the art will understand that Figure 7 The structure shown does not constitute a limitation on the computer device 700, and may include more or fewer components than shown, or combine certain components, or use different component arrangements.

[0081] In an exemplary embodiment, a computer-readable storage medium is also provided, such as a memory including instructions that can be executed by a processor in a terminal to complete the method for generating vocal audio in the above embodiments. This computer-readable storage medium can be non-transitory. For example, the computer-readable storage medium can be ROM (Read-Only Memory), RAM (Random Access Memory), CD-ROM (CompactDisc Read-Only Memory), magnetic tape, floppy disk, and optical data storage devices, etc.

[0082] In an exemplary embodiment, a computer program product is also provided, which includes program code that can be executed by a processor in a computer device to perform the method for generating singing audio described above.

[0083] It should be noted that all information (including but not limited to user device information, user personal information, etc.), data (including but not limited to data used for analysis, stored data, displayed data, etc.), and signals (including but not limited to signals transmitted between user terminals and other devices) involved in this application have been authorized by the user or fully authorized by all parties, and the collection, use, and processing of related data must comply with the relevant laws, regulations, and standards of the relevant countries and regions. For example, the page information involved in this application was obtained with full authorization.

[0084] Those skilled in the art will understand that all or part of the steps of the above embodiments can be implemented by hardware or by a program instructing related hardware. The program can be stored in a computer-readable storage medium, such as a read-only memory, a disk, or an optical disk.

[0085] It should be understood that "multiple" as used herein refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. Furthermore, the step numbers described herein are merely illustrative of one possible execution order. In some other embodiments, the steps may not be executed in numerical order, such as two steps with different numbers being executed simultaneously, or two steps with different numbers being executed in the reverse order of the illustration. This application does not limit this.

[0086] The above description is merely an optional embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. A method for generating singing audio, characterized in that, The method includes: Acquire the user's recorded audio and the singer's first performance audio of the target song; Extract the first timbre feature corresponding to the recorded audio and the singing feature corresponding to the first singing audio; A second timbre feature matching the first timbre feature is determined in a timbre library, wherein the timbre library includes timbre features corresponding to multiple singers; The first timbre feature is fused with the second timbre feature to obtain a fused timbre feature; Based on the fused timbre features and the singing features, a second singing audio of the user singing the target song is generated.

2. The method according to claim 1, characterized in that, The singing features include the melody features and semantic features corresponding to the first singing audio.

3. The method according to claim 1, characterized in that, The step of fusing the first timbre feature with the second timbre feature to obtain the fused timbre feature includes: Obtain the first weight coefficient corresponding to the first timbre feature and the second weight coefficient corresponding to the second timbre feature; The first timbre feature and the second timbre feature are weighted according to the first weighting coefficient and the second weighting coefficient to obtain the fused timbre feature.

4. The method according to claim 2, characterized in that, The step of weighting the first timbre feature and the second timbre feature according to the first weighting coefficient and the second weighting coefficient to obtain the fused timbre feature includes: Based on the first weighting coefficient and the second weighting coefficient, the first timbre feature and the second timbre feature are weighted and summed to obtain the third timbre feature; An average timbre feature is obtained, which is generated by a diffusion model under unrestricted input conditions. The diffusion model is used to generate the second singing audio based on the fused timbre feature, the melody feature, and the semantic feature. The third timbre feature is enhanced based on the average timbre feature to obtain the fused timbre feature.

5. The method according to claim 4, characterized in that, The enhancement process of the third timbre feature based on the average timbre feature to obtain the fused timbre feature includes: Obtain the enhancement coefficient; Determine the difference between the third timbre feature and the average timbre feature, and determine the product of the difference and the enhancement coefficient; The product of the third timbre feature and the product is summed to obtain the fused timbre feature.

6. The method according to claim 4, characterized in that, The step of generating a second vocal audio recording of the user singing the target song based on the fused timbre features and the singing features includes: The fused timbre features, the melody features, and the semantic features are input into the diffusion model, so that the diffusion model generates audio data that matches the fused timbre features based on the melody features and the semantic features; The audio data is input to a vocoder to obtain the second singing audio output by the vocoder.

7. The method according to any one of claims 1 to 6, characterized in that, The recorded audio is a third recording of a user singing the target song. Extracting the first timbre feature corresponding to the recorded audio includes: Obtain multiple audio frames included in a specified singing period in the third singing audio; Extract the fourth timbre feature corresponding to each audio frame from the plurality of audio frames; The fourth timbre features corresponding to the multiple audio frames are aggregated to obtain the first timbre feature corresponding to the recorded audio.

8. A computer device, characterized in that, The computer device includes a processor and a memory, the memory storing at least one instruction that is loaded and executed by the processor to perform the operations performed by the method for generating vocal audio as described in any one of claims 1 to 7.

9. A computer-readable storage medium, characterized in that, The storage medium stores at least one instruction, which is loaded and executed by a processor to perform the operations performed by the method for generating vocal audio as described in any one of claims 1 to 7.

10. A computer program product, characterized in that, The storage medium stores at least one piece of program code, which is loaded and executed by a processor to perform the operations of the method for generating vocal audio as described in any one of claims 1 to 8.

Images