Audio processing method, medium, apparatus, and computing device
By identifying and fine-tuning the target timbre features in the timbre transfer model library, the problems of low efficiency and high cost in AI cover singing are solved, achieving efficient and low-cost user timbre simulation and ensuring the accuracy and naturalness of the transferred human voice timbre.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- HANGZHOU NETEASE CLOUD MUSIC TECH CO LTD
- Filing Date
- 2024-12-05
- Publication Date
- 2026-06-05
AI Technical Summary
In existing AI cover singing technology, the efficiency of human voice transfer is low and the cost is high, making it difficult to accurately simulate the unique timbre characteristics of users.
By identifying the target timbre features in the timbre transfer model library and fine-tuning the pre-trained model, the audio to be replaced can be directly converted into the user's corresponding transferred voice, simplifying the model generation process.
It significantly reduces the cost and customization cycle of voice transfer, improves conversion efficiency, and ensures the accuracy and naturalness of the timbre of the transfer results.
Smart Images
Figure CN122157690A_ABST
Abstract
Description
Technical Field
[0001] The embodiments of this disclosure relate to the field of artificial intelligence technology, and more specifically, the embodiments of this disclosure relate to an audio processing method, medium, apparatus, and computing device. Background Technology
[0002] This section is intended to provide background or context for embodiments of this disclosure. The description herein is not intended to imply that it is prior art simply because it is included in this section.
[0003] Artificial Intelligence (AI) cover song technology uses artificial intelligence to reconstruct, generate, or simulate original music, producing cover songs with similar style and timbre.
[0004] AI can mimic the pitch, rhythm, and tone of an original singer, but accurately simulating a user's unique timbre (i.e., personalized voice characteristics) remains a challenge. Existing technologies rely on complex voiceprint recognition and generation models, often resulting in low conversion efficiency and high conversion costs. Summary of the Invention
[0005] This disclosure provides an audio processing method, medium, apparatus, and computing device to solve the technical problems of low efficiency and high cost in the implementation of human voice transfer in related technologies.
[0006] In a first aspect of the present disclosure, an audio processing method is provided, comprising: acquiring a user's audio to be processed and audio to be replaced;
[0007] A first timbre transfer model corresponding to the target timbre feature in the audio to be processed is determined from a preset timbre transfer model library. The timbre transfer model library contains timbre transfer models corresponding to different timbre feature ranges.
[0008] Based on the audio to be processed, the first timbre transfer model is trained to obtain the second timbre transfer model corresponding to the audio to be processed;
[0009] The first human voice corresponding to the audio to be replaced is input into the second timbre transfer model to obtain the user's corresponding transferred human voice;
[0010] The target audio is determined based on the migrated human voice, the first harmony and the first accompaniment corresponding to the audio to be replaced.
[0011] In one embodiment of this disclosure, determining a first timbre transfer model corresponding to the target timbre feature in the audio to be processed from a preset timbre transfer model library includes:
[0012] Determine the target timbre features in the audio to be processed;
[0013] The timbre feature range corresponding to the target timbre feature is determined in the timbre transfer model library;
[0014] The timbre transfer model corresponding to the timbre feature range where the target timbre feature is located is determined as the first timbre transfer model.
[0015] In one embodiment of this disclosure, before determining the target audio based on the migrated vocals, the first harmony, and the first accompaniment corresponding to the audio to be replaced, the method further includes:
[0016] The first voice is transposed in units of N octaves to obtain the first voice that can be restored in the high-frequency range by the first timbre transfer model, where N is a positive integer.
[0017] In one embodiment of this disclosure, determining the target audio based on the migrated vocals, the first harmony corresponding to the audio to be replaced, and the first accompaniment includes:
[0018] Based on a preset mixing template, the transferred vocals, the first harmony, and the first accompaniment are mixed to obtain the target audio. The mixing template is determined based on the rhythm information of the audio to be replaced.
[0019] In one embodiment of this disclosure, before mixing the transferred vocals, the first harmony, and the first accompaniment based on a preset mixing template to obtain the target audio, the method further includes:
[0020] The audio to be replaced is subjected to rhythm detection processing to obtain the rhythm information corresponding to the audio to be replaced.
[0021] The rhythm information is used as parameters of a preset deep learning model to construct the mixing template.
[0022] In one embodiment of this disclosure, before determining the first timbre transfer model corresponding to the target timbre feature in the audio to be processed from a preset timbre transfer model library, the method further includes:
[0023] The audio to be replaced is subjected to sibilance removal and noise reduction processing to obtain the optimized audio to be replaced;
[0024] And / or, perform sibilance removal and noise reduction processing on the audio to be processed to obtain optimized audio.
[0025] In one embodiment of this disclosure, before determining the first timbre transfer model corresponding to the target timbre feature in the audio to be processed from a preset timbre transfer model library, the method further includes:
[0026] The audio to be replaced is subjected to audio accompaniment separation processing to obtain the first human voice, the first harmony, and the first accompaniment.
[0027] In a second aspect of the present disclosure, a computer-readable storage medium is provided, characterized in that the computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, implement the method as described in any of the above embodiments.
[0028] In a third aspect of this disclosure, an audio processing apparatus is provided, comprising:
[0029] The acquisition module is used to acquire the user's audio to be processed and audio to be replaced;
[0030] The first determining module is used to determine, in a preset timbre transfer model library, a first timbre transfer model corresponding to the target timbre feature in the audio to be processed, wherein the timbre transfer model library contains timbre transfer models corresponding to different timbre feature ranges.
[0031] The second determining module is used to train the first timbre transfer model based on the audio to be processed, so as to obtain the second timbre transfer model corresponding to the audio to be processed.
[0032] The third determining module is used to input the first human voice corresponding to the audio to be replaced into the second timbre transfer model to obtain the user's corresponding transferred human voice.
[0033] The fourth determining module is used to determine the target audio based on the migrated human voice, the first harmony and the first accompaniment corresponding to the audio to be replaced.
[0034] In one embodiment of this disclosure, the first determining module is specifically used for:
[0035] Determine the target timbre features in the audio to be processed;
[0036] The timbre feature range corresponding to the target timbre feature is determined in the timbre transfer model library;
[0037] The timbre transfer model corresponding to the timbre feature range where the target timbre feature is located is determined as the first timbre transfer model.
[0038] In one embodiment of this disclosure, before determining the target audio based on the migrated vocals, the first harmony, and the first accompaniment corresponding to the audio to be replaced, the first determining module is further configured to:
[0039] The first voice is transposed in units of N octaves to obtain the first voice that can be restored in the high-frequency range by the first timbre transfer model, where N is a positive integer.
[0040] In one embodiment of this disclosure, the fourth determining module is specifically used for:
[0041] Based on a preset mixing template, the transferred vocals, the first harmony, and the first accompaniment are mixed to obtain the target audio. The mixing template is determined based on the rhythm information of the audio to be replaced.
[0042] In one embodiment of this disclosure, before the target audio is obtained by mixing the migrated vocals, the first harmony, and the first accompaniment based on a preset mixing template, the first determining module is further configured to:
[0043] The audio to be replaced is subjected to rhythm detection processing to obtain the rhythm information corresponding to the audio to be replaced.
[0044] The rhythm information is used as parameters of a preset deep learning model to construct the mixing template.
[0045] In one embodiment of this disclosure, before determining the first timbre transfer model corresponding to the target timbre feature in the audio to be processed from the preset timbre transfer model library, the first determining module is further configured to:
[0046] The audio to be replaced is subjected to sibilance removal and noise reduction processing to obtain the optimized audio to be replaced;
[0047] And / or, perform sibilance removal and noise reduction processing on the audio to be processed to obtain optimized audio.
[0048] In one embodiment of this disclosure, before determining the first timbre transfer model corresponding to the target timbre feature in the audio to be processed from the preset timbre transfer model library, the first determining module is further configured to:
[0049] The audio to be replaced is subjected to audio accompaniment separation processing to obtain the first human voice, the first harmony, and the first accompaniment.
[0050] In a fourth aspect of this disclosure, a computing device is provided, comprising: at least one processor;
[0051] and a memory communicatively connected to the at least one processor;
[0052] The memory stores instructions that can be executed by the at least one processor, which, when executed by the at least one processor, cause the computing device to perform the method described in any of the above embodiments.
[0053] According to the audio processing method, apparatus, medium, and computing device of this disclosure, the following steps can be taken: First, a first timbre transfer model corresponding to the target timbre feature in the audio to be processed is obtained from a timbre transfer model library; second, the first timbre transfer model is trained to obtain a second timbre transfer model; third, the first voice corresponding to the audio to be replaced is input into the second timbre transfer model to obtain a transferred voice; and fourth, the target audio is determined based on the transferred voice, the first harmony, and the first accompaniment corresponding to the audio to be replaced. By finding a pre-trained timbre transfer model from the user's audio to be processed and fine-tuning it, voice transfer can be achieved, significantly reducing costs and shortening the customization cycle, thus providing a better user experience. Attached Figure Description
[0054] The above and other objects, features, and advantages of this disclosure will become readily apparent from the following detailed description of exemplary embodiments, taken in conjunction with the accompanying drawings. Several embodiments of this disclosure are illustrated in the drawings by way of example and not limitation, in which:
[0055] Figure 1 A schematic diagram of a scenario for the audio processing method provided in this embodiment of the disclosure;
[0056] Figure 2 A flowchart of an audio processing method provided in an embodiment of this disclosure;
[0057] Figure 3 This is a schematic diagram of sound companion separation provided in an embodiment of the present disclosure;
[0058] Figure 4 This is a schematic diagram of the training process provided in an embodiment of the present disclosure;
[0059] Figure 5 A schematic diagram of the generation part of the mixing template provided in this embodiment of the disclosure;
[0060] Figure 6 A flowchart of an audio processing method provided in another embodiment of this disclosure;
[0061] Figure 7 A structural diagram of a storage medium provided in an embodiment of this disclosure;
[0062] Figure 8 This is a structural diagram of an audio processing apparatus provided in an embodiment of the present disclosure;
[0063] Figure 9 A structural diagram of a computing device provided in an embodiment of this disclosure;
[0064] In the accompanying drawings, the same or corresponding reference numerals indicate the same or corresponding parts. Detailed Implementation
[0065] The principles and spirit of this disclosure will now be described with reference to several exemplary embodiments. It should be understood that these embodiments are given merely to enable those skilled in the art to better understand and implement this disclosure, and are not intended to limit the scope of this disclosure in any way. Rather, these embodiments are provided to make this disclosure more thorough and complete, and to fully convey the scope of this disclosure to those skilled in the art.
[0066] Those skilled in the art will recognize that embodiments of this disclosure can be implemented as a system, apparatus, device, method, or computer program product. Therefore, this disclosure can be specifically implemented in the following forms: entirely hardware, entirely software (including firmware, resident software, microcode, etc.), or a combination of hardware and software.
[0067] According to embodiments of this disclosure, an audio processing method, apparatus, medium, and computing device are proposed.
[0068] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this disclosure are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use and processing of the relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation portals are provided for users to choose to authorize or refuse.
[0069] Furthermore, the number of any elements in the accompanying drawings is for illustrative purposes only and not for limitation, and any naming is for distinction only and has no limiting meaning.
[0070] The principles and spirit of this disclosure will be explained in detail below with reference to several representative embodiments. Invention Overview
[0072] The inventors have discovered that AI cover song technologies often rely on complex voiceprint recognition and generation models, which can lead to the following technical problems: low conversion efficiency and high conversion cost.
[0073] The following is a brief description using the open-source AI cover song technologies SO-VITS-SVC and RVC as examples:
[0074] SO-VITS-SVC has relatively low articulation and requires significant computational resources. Training a model for a single user's vocal cover requires 3-10 hours, with recording lengths exceeding one hour. Recording quality requirements include no background noise and clear, high-quality vocal recordings. RVC, on the other hand, requires relatively fewer computational resources, but at the cost of sacrificing timbre accuracy, resulting in relatively poor performance.
[0075] To address the aforementioned technical issues, this disclosure provides an audio processing method, medium, apparatus, and computing device. By setting multiple timbre transfer models corresponding to timbre feature ranges, when processing a user's audio, the user's timbre features can be extracted, the closest timbre transfer model can be directly determined, and fine-tuned. This fine-tuned model can then be directly used to process the audio to be replaced, obtaining the user's corresponding transferred vocals. This allows for the rapid replacement of music audio with music audio corresponding to the transferred vocals. In this process, a complex model generation process is unnecessary, reducing implementation costs.
[0076] After introducing the basic principles of this disclosure, various non-limiting embodiments of this disclosure will be described in detail below.
[0077] Application Scenarios Overview
[0078] Firstly, the audio processing method provided in this disclosure can be applied to… Figure 1 The scene diagram shown. Figure 1 This is a schematic diagram illustrating a scenario of the audio processing method provided in an embodiment of this disclosure. Figure 1 As shown, the scenario includes: the user's recorded audio 11, music audio 12, and the replaced music audio 13.
[0079] In this application scenario, the computing device extracts the timbre features from the recorded audio 11, determines the timbre transfer model corresponding to the timbre in the timbre transfer model library, trains the timbre transfer model based on the recorded audio 11, and then inputs the recorded audio 11 into the trained timbre transfer model to obtain the transferred human voice.
[0080] Furthermore, the computing device performs vocal separation on the music audio 12 to obtain harmony and accompaniment, and then generates a replacement music audio 13 based on the transferred vocals, which the user can then use as the music audio for their own singing.
[0081] It should be noted that, Figure 1 This is merely a schematic diagram illustrating one application scenario provided by an embodiment of this disclosure; this embodiment does not necessarily represent an application scenario. Figure 1 The equipment included is limited.
[0082] Exemplary methods
[0083] The following is combined with Figure 1 The above application scenarios are used to describe the methods used according to exemplary embodiments of this disclosure. It should be noted that the above application scenarios are shown only to facilitate understanding of the spirit and principles of this disclosure, and the embodiments of this disclosure are not limited in any way. Rather, the embodiments of this disclosure can be applied to any applicable scenario.
[0084] Figure 2 This is a flowchart illustrating an audio processing method according to an embodiment of the present disclosure. The method described in this embodiment can be applied to a computing device, which may be a server or a terminal device. Figure 2 As shown, the method in this embodiment includes:
[0085] S21. Obtain the user's audio to be processed and audio to be replaced;
[0086] In this step, when a user needs to convert the audio to be replaced into a song generated by the user's own voice, the electronic device first obtains the user's audio, i.e. the audio to be processed, and the audio to be replaced.
[0087] In one possible implementation, the audio to be processed can be a user's recording, etc.; the audio to be replaced can be music that the user wants to transfer vocals to.
[0088] Optionally, before step 22 below, the audio to be replaced may also be subjected to sibilance removal and noise reduction processing to obtain optimized audio to be replaced; and / or, the audio to be processed may also be subjected to sibilance removal and noise reduction processing to obtain optimized audio to be processed.
[0089] In this possible implementation, sibilance refers to the sharp sound produced by certain consonants during pronunciation. These syllables are usually high-frequency and can produce an unpleasant harshness during recording, especially when the user or singer pronounces them too strongly. The goal of sibilance removal processing is to reduce or eliminate these unpleasant high-frequency "harsh" sounds, making the audio sound smoother and more natural. The purpose of noise reduction processing is to remove background noise or unwanted noise from the audio, such as electrical noise during recording, wind noise, keyboard typing, ambient noise, etc.
[0090] For example, de-essences can be used to automatically detect and suppress excessive essences; noise analysis, noise suppression, and frequency correction can be used to remove noise.
[0091] Optionally, after step 21, the audio to be replaced may also be subjected to audio-accompaniment separation processing to obtain the first vocal, the first harmony, and the first accompaniment.
[0092] In this possible implementation, the harmonies and orchestrations in most finished songs are complex, and common vocal separation models struggle to obtain ideal vocal segments. This disclosure can combine open-source technologies, such as U-Net and Transformer, to chain together multiple model structures, thereby leveraging the strengths of different models to separate the finished song into three parts: lead vocals (first vocals), harmonies (first harmony), and accompaniment (first accompaniment).
[0093] This provides a master vocal track with a single pitch for the subsequent timbre transfer model, and provides harmony and accompaniment tracks for the implementation in step 25.
[0094] In one possible implementation, the audio to be replaced is a song, and the audio to be processed is a recording:
[0095] Figure 3 This is a schematic diagram of sound companion separation provided in an embodiment of the present disclosure, as shown below. Figure 3 As shown, the schematic diagram includes: Figure 3 The upper part of the recording is separated into sound and sound. Figure 3 The second part of the song features a separate vocal accompaniment.
[0096] For audio recordings, the system separates the main vocals, harmonies, and accompaniment; removes silent segments; removes reverb; restores sound quality (removing breathing sounds, sibilance, popping sounds, etc.); equalizes volume; and reduces noise.
[0097] For song audio separation, extract the lead vocals, harmonies, accompaniment, etc.; remove reverb; restore sound quality (breathing sounds, sibilance, popping sounds, etc.); equalize volume; and reduce noise.
[0098] The significance of the above implementation lies in the fact that, during testing, the inventors discovered that due to the varying quality of users' recording equipment and environments, the recorded materials often have poor sound quality and, in most cases, contain environmental noise and other sound quality issues. If these unprocessed recording materials are directly used for model training, the transfer results may exhibit defects such as noise and current noise, thus affecting the overall performance.
[0099] Therefore, in addition to the necessary background noise removal and volume equalization steps, audio quality restoration modules such as sibilance removal and noise reduction have been added to the training data processing flow. These additional processing steps help to further improve the audio quality of the recording materials. Furthermore, the recording materials can be thoroughly screened and filtered to remove segments with obvious risks, ensuring high-quality training data. Through these improvements, the training quality of the user voice transfer model has been significantly enhanced, ensuring that the transfer results are purer and more natural.
[0100] S22. Determine the first timbre transfer model corresponding to the target timbre feature in the audio to be processed from the preset timbre transfer model library;
[0101] In this step, the preset timbre transfer model library contains timbre transfer models corresponding to different timbre feature ranges. Each timbre transfer model corresponding to a timbre feature range can be trained based on different recording materials and other training data, and can convert the input recording into a human voice within the preset timbre range.
[0102] That is, the timbre transfer model can transfer the timbre features of the original human voice to the target timbre features. In this embodiment of the disclosure, the human voice in the audio to be replaced is converted into the human voice corresponding to the user.
[0103] In one possible implementation, the timbre transfer model could be based on the VITS architecture, consisting of a text encoder, WavNet (for high-fidelity TTS), Hifi-GAN (for converting voice features into high-fidelity speech waveforms), and other structures.
[0104] In one possible implementation, Figure 4 This is a schematic diagram of the training process provided in an embodiment of the present disclosure. The first part of the process is as follows:
[0105] By using available voice and singing data, such as open-source voice and singing data from the Internet, in an offline process, various models for generating different timbre transfers are trained to build a "timbre transfer model library".
[0106] S23. Based on the audio to be processed, train the first timbre transfer model to obtain the second timbre transfer model corresponding to the audio to be processed.
[0107] In this step, in order to make the processing result of the first timbre transfer model more closely match the human voice in the audio to be processed, the first timbre transfer model is trained based on the audio to be processed to obtain the second timbre transfer model corresponding to the audio to be processed.
[0108] In one possible implementation, based on Figure 4 The second step in the process implementation is as follows:
[0109] Based on the "timbre transfer model library", after determining the first timbre transfer model, fine-tuning training is performed using user recording samples. The model can converge quickly within a few iterations, keeping the training time within a few minutes (e.g., 4 minutes).
[0110] That is, the second timbre transfer model = screening of the pre-trained timbre transfer model library + weighted fusion (i.e. fine-tuning). For example, after a set of models based on the user's feature index (timbre features), a similar timbre model is fused, and then fine-tuned on this basis.
[0111] exist Figure 4In the "training pipeline" and "production pipeline," the process proceeds sequentially from left to right. The first stage is "feature extraction," which involves extracting useful features from the audio to be processed using a lower sampling rate and selecting a first timbre transfer model. Then, higher sampling rate audio data is processed and quickly fine-tuned to obtain a second timbre transfer model. The first voice corresponding to the audio to be replaced is input into the second timbre transfer model to obtain the changed timbre, i.e., the transferred voice corresponding to the user.
[0112] Among them, top1 retrieval (weighted mixing) replaces the input source features with training set features to eliminate timbre leakage.
[0113] S24. Input the first human voice corresponding to the audio to be replaced into the second timbre transfer model to obtain the user's corresponding transferred human voice;
[0114] In this step, the first human voice in the audio to be replaced is input into the second timbre transfer model trained above, which can obtain the human voice in the audio to be replaced by the user's human voice, that is, the transferred human voice.
[0115] Optionally, before step 25 below, the following can also be performed: transpose the first voice in units of N octaves to obtain the first voice in the high-frequency range that the first timbre transfer model can restore, where N is a positive integer.
[0116] In this implementation, the training process of the aforementioned timbre transfer model suffers from a significant difference in pitch range between the target song and the training data, often resulting in unsatisfactory transfer effects. This is caused by differences in vocal ranges between different singers (especially male and female singers) due to both innate and acquired factors, leading to substantial pitch differences between different songs. In practical applications, this can easily result in a mismatch between the song and the singer's vocal range.
[0117] Furthermore, recordings are typically short, of poor quality, and unpredictable, which severely limits the pitch range available for training data. For data-driven deep models, it is difficult to reproduce the proper timbre of vocals that exceed the pitch range of the training data.
[0118] Therefore, in timbre transfer, the pitch range of the audio to be replaced can be shifted to the range that the timbre transfer model can reproduce by adjusting the pitch of the first vocal. This is known as "free transposition." However, during mixing, to ensure a harmonious match between the accompaniment and the transferred vocal, the accompaniment needs to undergo the same pitch adjustment, which can lead to distortion. To solve this problem, the vocal can be transposed in N octaves without transposing the accompaniment. In this way, while maintaining the high fidelity of the accompaniment, the mixed result also achieves a harmonious and unified sound.
[0119] S25. Determine the target audio based on the first harmony and first accompaniment corresponding to the migrated vocals and the audio to be replaced.
[0120] In this step, after obtaining the migrated vocals, the timbre characteristics and acoustic properties of the migrated vocals are analyzed. Combined with the processing algorithms of various audio effects processors, the user's vocals after migration are mixed with the first harmony and the first accompaniment in the audio to be replaced to obtain the target audio.
[0121] Optionally, one implementation of step 25 could be: based on a preset mixing template, perform mixing processing on the transferred vocals, first harmony, and first accompaniment to obtain the target audio.
[0122] The mixing template is determined based on the tempo information (Beats Per Minute, BPM) of the audio to be replaced.
[0123] In one possible implementation, the mixing template could be: performing rhythm detection processing on the audio to be replaced to obtain the rhythm information corresponding to the audio to be replaced; and using the rhythm information as parameters of a preset deep learning model to construct the mixing template.
[0124] In one possible implementation, the mixing process can consist of a processing chain comprising four key steps: algorithm analysis, vocal processing, volume balancing, and mastering. The effects used include compressors, parametric EQ, dynamic EQ, multiband exciter, reverb, and delay.
[0125] To achieve accurate mixing, the prior techniques of professional mixing engineers can be leveraged, combined with the characteristics of different songs, as well as methods such as metric learning and acoustic analysis, to precisely calculate the optimal processing parameters for each effect unit. To ensure that the final product achieves a release-ready listening experience, the automatic mastering algorithm comprehensively enhances the overall loudness, dynamics, and soundstage of the mixed audio to market-ready levels, and produces the final song.
[0126] Figure 5 This is a schematic diagram of the generation part of the mixing template provided in the embodiments of this disclosure, as shown below. Figure 5 As shown, the dry sound is based on the corrected parameters, and the corresponding wet sound is assigned a weight to the parameters to obtain the wet sound.
[0127] As a crucial attribute of a song, rhythm information directly influences a user's perception of its style. To achieve high-quality automatic mixing, the rhythm information of the finished song can be used as an important parameter in the construction of mixing templates.
[0128] Furthermore, the rhythm information detection model used in this embodiment can be developed and constructed based on the Transformer structure.
[0129] This disclosure provides an audio processing method, comprising: acquiring a user's audio to be processed and an audio to be replaced; determining a first timbre transfer model corresponding to a target timbre feature in the audio to be processed from a timbre transfer model library; training the first timbre transfer model to obtain a second timbre transfer model; inputting a first voice corresponding to the audio to be replaced into the second timbre transfer model to obtain a transferred voice; and determining the target audio based on the transferred voice, a first harmony, and a first accompaniment corresponding to the audio to be replaced. In this technical solution, by setting multiple timbre transfer models corresponding to timbre feature ranges, when processing a user's audio, the user's timbre features can be extracted, the closest timbre transfer model can be directly determined, and fine-tuned so that the fine-tuned model can be directly used to process the audio to be replaced to obtain the user's corresponding transferred voice, thereby quickly replacing music audio with music audio corresponding to the transferred voice. In this process, a complex model generation process is unnecessary, reducing implementation costs.
[0130] based on Figure 2 The embodiment shown will now be further described in detail below. Figure 6 A flowchart illustrating an audio processing method provided in another embodiment of this disclosure. Figure 6 As shown, S22 may include the following steps:
[0131] S61. Determine the target timbre features in the audio to be processed;
[0132] In this step, based on the user's audio to be processed, the timbre feature data, i.e., the target timbre feature, can be extracted from the audio.
[0133] In one possible implementation, the frequency, transients, loudness, and other characteristics of the audio to be processed can be comprehensively analyzed through methods such as spectrum analysis, time-domain analysis, dynamic range, harmonic structure, and formants to obtain the target timbre characteristics.
[0134] S62. Determine the range of timbre features that correspond to the target timbre features in the timbre transfer model library;
[0135] In this step, the timbre transfer model library stores timbre transfer models corresponding to different timbre feature ranges. After obtaining the target timbre feature, the timbre feature range in which the target timbre feature is located can be determined based on the target timbre feature in different timbre feature ranges.
[0136] S63. The timbre transfer model corresponding to the timbre feature range where the target timbre feature is located is determined as the first timbre transfer model.
[0137] This disclosure provides an audio processing method, comprising: determining a target timbre feature in the audio to be processed; determining a timbre feature range in a timbre transfer model library that corresponds to the target timbre feature; and determining the timbre transfer model corresponding to the timbre feature range of the target timbre feature as a first timbre transfer model. In this technical solution, by utilizing the target timbre feature in the audio to be processed, a transfer model with a similar timbre is quickly determined in the timbre transfer model library, thereby improving the efficiency of subsequent audio determination.
[0138] Exemplary media
[0139] After introducing the methods of exemplary embodiments of this disclosure, the following references are made. Figure 7 The storage medium of the exemplary embodiments of this disclosure will be described.
[0140] Figure 7 This is a structural diagram of a storage medium provided in an embodiment of the present disclosure, with reference to... Figure 7 As shown, the storage medium 70 stores a program product for implementing the above-described method according to embodiments of the present disclosure. This program product may be a portable compact disc read-only memory (CD-ROM) and includes program code, and may run on a computing device, such as a personal computer. However, the program product of the present disclosure is not limited thereto.
[0141] The program product may employ any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: an electrical connection having one or more wires, a portable disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.
[0142] A readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, carrying readable program code. This propagated data signal may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A readable signal medium may also be any readable medium other than a readable storage medium.
[0143] Program code for performing the operations disclosed herein can be written in any combination of one or more programming languages, including object-oriented programming languages such as Java and C++, and conventional procedural programming languages such as C or similar languages. The program code can execute entirely on the user's computing device, partially on the user's computing device, partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing devices can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN).
[0144] Exemplary device
[0145] Having introduced the medium of exemplary embodiments of this disclosure, the following references are made to... Figure 8 The audio processing apparatus of the exemplary embodiments of this disclosure is described to implement the method in any of the above method embodiments. The implementation principle and technical effect are similar, and will not be repeated here. Figure 8 This is a structural diagram of an audio processing apparatus provided according to an embodiment of the present disclosure. Figure 8 As shown, the audio processing device includes:
[0146] Module 81 is used to acquire the user's audio to be processed and audio to be replaced;
[0147] The first determining module 82 is used to determine the first timbre transfer model corresponding to the target timbre feature in the audio to be processed from the preset timbre transfer model library. The timbre transfer model library contains timbre transfer models corresponding to different timbre feature ranges.
[0148] The second determining module 83 is used to train the first timbre transfer model based on the audio to be processed, so as to obtain the second timbre transfer model corresponding to the audio to be processed.
[0149] The third determining module 84 is used to input the first human voice corresponding to the audio to be replaced into the second timbre transfer model to obtain the user's corresponding transferred human voice.
[0150] The fourth determining module 85 is used to determine the target audio based on the first harmony and first accompaniment corresponding to the migrated vocals and the audio to be replaced.
[0151] In one embodiment of this disclosure, the first determining module 82 is specifically used for:
[0152] Determine the target timbre features in the audio to be processed;
[0153] Determine the range of timbre features that correspond to the target timbre features in the timbre transfer model library;
[0154] The timbre transfer model corresponding to the timbre feature range where the target timbre feature is located is determined as the first timbre transfer model.
[0155] In one embodiment of this disclosure, before determining the target audio based on the migrated vocals, the first harmony, and the first accompaniment corresponding to the audio to be replaced, the first determining module 82 is further configured to:
[0156] The first voice is transposed in units of N octaves to obtain the first voice that can be restored in the high-frequency range by the first timbre transfer model, where N is a positive integer.
[0157] In one embodiment of this disclosure, the fourth determining module 85 is specifically used for:
[0158] Based on a preset mixing template, the transferred vocals, first harmony, and first accompaniment are mixed to obtain the target audio. The mixing template is determined based on the rhythm information of the audio to be replaced.
[0159] In one embodiment of this disclosure, before mixing the transferred vocals, first harmony, and first accompaniment based on a preset mixing template to obtain the target audio, the first determining module 81 is further configured to:
[0160] Perform rhythm detection processing on the audio to be replaced to obtain the rhythm information corresponding to the audio to be replaced;
[0161] Rhythm information is used as parameters of a pre-defined deep learning model to construct a mixing template.
[0162] In one embodiment of this disclosure, before determining a first timbre transfer model corresponding to the target timbre feature in the audio to be processed from a preset timbre transfer model library, the first determining module 81 is further configured to:
[0163] The audio to be replaced is processed by sibilance removal and noise reduction to obtain the optimized audio to be replaced.
[0164] And / or, perform sibilance removal and noise reduction processing on the audio to be processed to obtain the optimized audio.
[0165] In one embodiment of this disclosure, before determining a first timbre transfer model corresponding to the target timbre feature in the audio to be processed from a preset timbre transfer model library, the first determining module 81 is further configured to:
[0166] The audio to be replaced is processed by audio-accompaniment separation to obtain the first vocal, the first harmony, and the first accompaniment.
[0167] Exemplary computing device
[0168] Having described the methods, media, and apparatus of exemplary embodiments of this disclosure, the following references... Figure 9 A computing device according to an exemplary embodiment of the present disclosure will be described.
[0169] Figure 9 This is a structural diagram of a computing device provided in an embodiment of the present disclosure. Figure 9 The computing device shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments disclosed herein.
[0170] like Figure 9 As shown, the computing device is presented in the form of a general-purpose computing device. The components of the computing device may include, but are not limited to: at least one processing unit 101, at least one storage unit 102, and a bus 103 connecting different system components (including the processing unit 101 and the storage unit 102). The at least one storage unit 102 stores computer-executable instructions; the at least one processing unit 101 includes a processor that executes the computer-executable instructions to implement the method described above.
[0171] Bus 103 includes a data bus, a control bus, and an address bus.
[0172] Storage unit 102 may include readable media in the form of volatile memory, such as random access memory (RAM) 1021 and / or cache memory 1022, and may further include readable media in the form of non-volatile memory, such as read-only memory (ROM) 1023.
[0173] Storage unit 102 may also include a program / utility 1025 having a set (at least one) program module 1024, such program module 1024 including but not limited to: operating system, one or more application programs, other program modules and program data, each of these examples or some combination of these may include an implementation of a network environment.
[0174] The computing device can also communicate with one or more external devices 104 (e.g., keyboard, pointing device, etc.). This communication can be performed via input / output (I / O) interface 105. Furthermore, the computing device can also communicate with one or more networks (e.g., local area network (LAN), wide area network (WAN), and / or public networks, such as the Internet) via network adapter 106. Figure 9 As shown, network adapter 106 communicates with other modules of the computing device via bus 103. It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in conjunction with the computing device, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.
[0175] It should be noted that although several units / modules or sub-units / modules of the audio processing device have been mentioned in the detailed description above, this division is merely exemplary and not mandatory. In fact, according to embodiments of this disclosure, the features and functions of two or more units / modules described above can be embodied in one unit / module. Conversely, the features and functions of one unit / module described above can be further divided and embodied by multiple units / modules.
[0176] Furthermore, although the operations of the methods disclosed herein are described in a specific order in the accompanying drawings, this does not require or imply that these operations must be performed in that specific order, or that all of the operations shown must be performed to achieve the desired result. Additionally or alternatively, certain steps may be omitted, multiple steps may be combined into one step, and / or one step may be broken down into multiple steps.
[0177] While the spirit and principles of this disclosure have been described with reference to several specific embodiments, it should be understood that this disclosure is not limited to the disclosed specific embodiments, and the division of aspects does not imply that features in these aspects cannot be combined for benefit; such division is merely for convenience of expression. This disclosure is intended to cover various modifications and equivalent arrangements included within the spirit and scope of the appended claims.
Claims
1. An audio processing method, characterized in that, include: Obtain the user's audio files to be processed and audio files to be replaced; A first timbre transfer model corresponding to the target timbre feature in the audio to be processed is determined from a preset timbre transfer model library. The timbre transfer model library contains timbre transfer models corresponding to different timbre feature ranges. Based on the audio to be processed, the first timbre transfer model is trained to obtain the second timbre transfer model corresponding to the audio to be processed; The first human voice corresponding to the audio to be replaced is input into the second timbre transfer model to obtain the user's corresponding transferred human voice; The target audio is determined based on the migrated human voice, the first harmony and the first accompaniment corresponding to the audio to be replaced.
2. The method according to claim 1, characterized in that, The step of determining the first timbre transfer model corresponding to the target timbre features in the audio to be processed from the preset timbre transfer model library includes: Determine the target timbre features in the audio to be processed; The timbre feature range corresponding to the target timbre feature is determined in the timbre transfer model library; The timbre transfer model corresponding to the timbre feature range where the target timbre feature is located is determined as the first timbre transfer model.
3. The method according to claim 1 or 2, characterized in that, Before determining the target audio based on the migrated vocals, the first harmony, and the first accompaniment corresponding to the audio to be replaced, the method further includes: The first voice is transposed in units of N octaves to obtain the first voice that can be restored in the high-frequency range by the first timbre transfer model, where N is a positive integer.
4. The method according to claim 1 or 2, characterized in that, The step of determining the target audio based on the migrated human voice, the first harmony and the first accompaniment corresponding to the audio to be replaced includes: Based on a preset mixing template, the transferred vocals, the first harmony, and the first accompaniment are mixed to obtain the target audio. The mixing template is determined based on the rhythm information of the audio to be replaced.
5. The method according to claim 4, characterized in that, Before mixing the transferred vocals, the first harmony, and the first accompaniment based on a preset mixing template to obtain the target audio, the method further includes: The audio to be replaced is subjected to rhythm detection processing to obtain the rhythm information corresponding to the audio to be replaced. The rhythm information is used as parameters of a preset deep learning model to construct the mixing template.
6. The method according to claim 1 or 2, characterized in that, Before determining the first timbre transfer model corresponding to the target timbre feature in the audio to be processed from the preset timbre transfer model library, the method further includes: The audio to be replaced is subjected to sibilance removal and noise reduction processing to obtain the optimized audio to be replaced; And / or, perform sibilance removal and noise reduction processing on the audio to be processed to obtain optimized audio.
7. The method according to claim 1 or 2, characterized in that, Before determining the first timbre transfer model corresponding to the target timbre feature in the audio to be processed from the preset timbre transfer model library, the method further includes: The audio to be replaced is subjected to audio accompaniment separation processing to obtain the first human voice, the first harmony, and the first accompaniment.
8. An audio processing apparatus, characterized in that, include: The acquisition module is used to acquire the user's audio to be processed and audio to be replaced; The first determining module is used to determine, in a preset timbre transfer model library, a first timbre transfer model corresponding to the target timbre feature in the audio to be processed, wherein the timbre transfer model library contains timbre transfer models corresponding to different timbre feature ranges. The second determining module is used to train the first timbre transfer model based on the audio to be processed, so as to obtain the second timbre transfer model corresponding to the audio to be processed. The third determining module is used to input the first human voice corresponding to the audio to be replaced into the second timbre transfer model to obtain the user's corresponding transferred human voice. The fourth determining module is used to determine the target audio based on the migrated human voice, the first harmony and the first accompaniment corresponding to the audio to be replaced.
9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, implement the method as described in any one of claims 1 to 7.
10. A computing device, characterized in that, include: At least one processor; and a memory communicatively connected to the at least one processor; The memory stores instructions executable by the at least one processor, which, when executed by the at least one processor, cause the computing device to perform the method as described in any one of claims 1 to 7.