Timbre Generation Methods and Systems
By acquiring fine-grained features of timbre, pitch, and instrument type, and using neural networks to fuse these features and generate Mel spectrograms, the problem of lack of fine-grained control in timbre generation in existing technologies is solved, and personalized customization of timbre is achieved.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- SHANGHAI BILIBILI TECH CO LTD
- Filing Date
- 2023-07-18
- Publication Date
- 2026-05-26
AI Technical Summary
Existing timbre generation methods lack fine-grained control, making it difficult to accurately customize timbre according to the personalized needs of music producers.
By acquiring fine-grained timbre features, pitch features, and instrument type features, feature fusion is performed using a neural network, and a modified ResUnet++ model is used for denoising diffusion to generate a Mel spectrogram. Finally, the target audio is generated through a vocoder.
It enables fine-grained, controllable generation of timbre, breaking down the barrier between perceptual description and feature values, and supporting music producers in creating personalized timbres.
Smart Images
Figure CN116758882B_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of sound processing technology, and in particular to a timbre generation method, system, electronic device, and computer-readable storage medium. Background Technology
[0002] With the development of computer technology and the increasing diversification of popular music aesthetics, timbre is playing an increasingly important role in music. The trend towards digital music production processes has freed the choice of timbre from traditional instruments, and more and more music producers are using electronic instruments and software sound sources to create timbres that match the desired characteristics of the songs.
[0003] Previous methods of timbre generation mostly used instrument type or pitch as labels to generate or transfer timbre for a specific type of instrument. The main research goal was instrument-level generation, rather than finer-grained control. This has led to a disconnect between research content and actual industry needs. Music producers with different skill levels need personalized timbre customization based on their needs, rather than simply selecting a particular type of instrument. Summary of the Invention
[0004] The main purpose of this application is to propose a timbre generation method, system, electronic device, and computer-readable storage medium, aiming to solve the problem of how to generate more fine-grained and controllable timbres.
[0005] To achieve the above objectives, embodiments of this application provide a timbre generation method, the method comprising:
[0006] Obtain the required fine-grained timbre features, pitch features, and instrument type features;
[0007] The fine-grained timbre features, the pitch features, and the instrument type features are fused together.
[0008] The fused feature data is input into the trained generative model for denoising and diffusion to obtain the Mel spectrogram of the target timbre.
[0009] The Mel spectrogram is processed by a vocoder to generate the audio of the target timbre.
[0010] Optionally, the acquisition of fine-grained timbre features includes:
[0011] Provide users with preset timbre descriptors or descriptor pairs, and a numerical range representing the degree of each descriptor or descriptor pair, wherein each descriptor or descriptor pair corresponds to a timbre feature;
[0012] Obtain the numerical value selected by the user for each of the descriptive words or descriptive word pairs, and combine the descriptive words or descriptive word pairs and their corresponding numerical values to obtain the fine-grained timbre feature.
[0013] Optionally, the acquisition of instrument type characteristics includes:
[0014] Provides a dataset of musical instrument types;
[0015] Obtain the instrument category label selected by the user based on the instrument category dataset;
[0016] The corresponding instrument type characteristics are obtained based on the instrument type label.
[0017] Optionally, providing the user with preset timbre descriptive words or pairs of descriptive words, and the numerical range of each of the descriptive words or pairs of descriptive words, includes:
[0018] Provide users with timbre descriptions including hardness, depth, brightness, roughness, warmth, sharpness, and low-frequency resonance, and set the numerical range of each description to 1 to 100.
[0019] Optionally, the feature fusion of the fine-grained timbre feature, the pitch feature, and the instrument type feature includes:
[0020] The pitch feature and the instrument type feature are concatenated, and the concatenated feature is fused with the time step information in the generation model;
[0021] The fine-grained timbre features are spliced together with the noise to be denoised.
[0022] Optionally, the step of inputting the fused feature data into the trained generative model for denoising and diffusion to obtain the Mel spectrogram of the target timbre includes:
[0023] The denoising diffusion generation model is input with Gaussian noise whose head is concatenated with the fine-grained timbre feature information, and the embedding vector pointing to the time step contains the pitch feature and the instrument type feature information, to obtain the Mel spectrum of the target timbre output by the denoising diffusion generation model.
[0024] Optionally, the generative model uses a bilateral denoising diffusion model for inference optimization to shorten the reverse inference process.
[0025] Optionally, the generative model uses an improved ResUnet++ model as a denoising network.
[0026] Furthermore, to achieve the above objectives, embodiments of this application also provide a timbre generation system, the system comprising:
[0027] The acquisition module is used to acquire the required fine-grained timbre features, pitch features, and instrument type features;
[0028] The fusion module is used to fuse the fine-grained timbre features, the pitch features, and the instrument type features.
[0029] The denoising and diffusion module is used to input the fused feature data into the trained generative model for denoising and diffusion, and to obtain the Mel spectrogram of the target timbre.
[0030] The processing module is used to process the Mel spectrogram through a vocoder to generate audio of the target timbre.
[0031] To achieve the above objectives, this application also provides an electronic device, which includes: a memory, a processor, and a timbre generation program stored in the memory and executable on the processor. When the timbre generation program is executed by the processor, it implements the timbre generation method as described above.
[0032] To achieve the above objectives, embodiments of this application also provide a computer-readable storage medium storing a timbre generation program, which, when executed by a processor, implements the timbre generation method as described above.
[0033] The timbre generation method, system, electronic device, and computer-readable storage medium proposed in this application can utilize the inductive ability of neural networks and the generative ability of generative models to generate timbre, and integrate features of three different scales—instrument type, pitch, and fine-grained timbre description information—to perform fine-grained control on the desired timbre. Attached Figure Description
[0034] Figure 1 This is a schematic diagram illustrating an application environment for implementing various embodiments of this application;
[0035] Figure 2 This is a flowchart of a timbre generation method proposed in the first embodiment of this application;
[0036] Figure 3 for Figure 2 A schematic diagram of a refinement process for obtaining fine-grained timbre features in step S200;
[0037] Figure 4 for Figure 2 A detailed flowchart illustrating the process of obtaining instrument type characteristics in step S200;
[0038] Figure 5 This is a schematic diagram of a feature fusion method based on the first embodiment;
[0039] Figure 6 This is a flowchart illustrating another form of the timbre generation method proposed in the first embodiment;
[0040] Figure 7 This is a schematic diagram of the hardware architecture of an electronic device according to the second embodiment of this application;
[0041] Figure 8 This is a schematic diagram of a timbre generation system proposed in the third embodiment of this application. Detailed Implementation
[0042] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application. All other embodiments obtained by those skilled in the art based on the embodiments in this application without inventive effort are within the scope of protection of this application.
[0043] It should be noted that the descriptions involving "first," "second," etc., in the embodiments of this application are for descriptive purposes only and should not be construed as indicating or implying their relative importance or implicitly specifying the number of technical features indicated. Therefore, a feature defined with "first" or "second" may explicitly or implicitly include at least one of that feature. Furthermore, the technical solutions of the various embodiments can be combined with each other, but this must be based on the ability of those skilled in the art to implement them. If the combination of technical solutions is contradictory or impossible to implement, it should be considered that such a combination of technical solutions does not exist and is not within the scope of protection claimed in this application.
[0044] The following are explanations of terms that may be involved in the embodiments of this application:
[0045] Timbre: refers to the tone quality or timbre of a musical instrument, which is the difference in auditory attributes displayed by two sounds with the same loudness and pitch. As one of the four elements of music, timbre differs from characteristics such as dynamics and pitch, which have certain objective indicators, and relies more on human hearing and subjective perception.
[0046] CLIP (Contrastive Language–Image Pre-training): This method uses supervised signals from text to train a visual pre-trained model with strong transfer capabilities. Through contrastive learning, it trains to obtain the similarity (association) between images and text. CLIP is primarily used for image classification, but it can also assist in GAN (Generative Animation Programming) algorithms, detection, segmentation, and retrieval.
[0047] DDPM (Denoising Diffusion Probabilistic Models) is a denoising diffusion probabilistic model. Simply put, this model consists of two processes: a forward diffusion process and a backward generation process. The forward diffusion process gradually adds Gaussian noise to an image until it becomes random noise, while the backward generation process is the denoising process, which starts with random noise and gradually removes it until an image is generated; this is the part that needs to be trained.
[0048] BDDM (Bilateral Denoising Diffusion Models): A novel bilateral denoising diffusion model that generates high-quality samples in far fewer steps than DDPM. From a bilateral modeling perspective, BDDM parameterizes the forward and backward processes using scoring and scheduling networks, respectively. BDDM is efficient, easy to train, and can further improve any pre-trained DDPM by optimizing inference noise scheduling.
[0049] The ResUnet++ model is a semantic segmentation neural network, originally an improved ResUnet architecture for colonoscopy image segmentation. ResUnet is a deep convolutional neural network structure. This application modifies the original ResUnet++ model for use in denoising and diffusion during timbre generation.
[0050] Mel spectrogram: A spectrogram in which frequencies are converted to a mel scale. A mel scale is a unit of pitch where equal intervals are perceived as equal by the listener.
[0051] Vocoder: Also known as a speech analysis and synthesis system. At the transmitting end, the speech signal is analyzed, its characteristic parameters are extracted, encoded, and encrypted to match the channel. This information is then transmitted to the receiving end, where the original speech waveform is reconstructed based on the received characteristic parameters.
[0052] Previous methods of timbre generation mostly used instrument type or pitch as labels to generate or transfer the timbre of a certain type of instrument, lacking research on more granular control.
[0053] Recently, with the emergence of multimodal models like CLIP, some schemes have emerged that utilize latent space representations of labeled text information and timbre itself within the dataset for matching, attempting to use subjective descriptive words to achieve finer-grained control over a particular type of timbre. However, the effectiveness of this method heavily relies on the accuracy of the descriptive words, and high-quality human timbre descriptions have several problems: the inherently subjective nature of timbre can lead to subjective results. To meet the data requirements of multimodal models, a large-scale timbre-label pair needs to be created, a task that requires significant manpower and time. This descriptive method cannot reflect finer-grained "degree" features; for example, two timbre descriptions might both be "dark," but there's no way to distinguish the degree of their features using a single descriptive word.
[0054] Therefore, based on the above-mentioned technical deficiencies, this application proposes a new timbre generation scheme, which provides a more objective and unified description of timbre, and utilizes the inductive ability of neural networks and the generative ability of generative models to generate timbre. In the generation process, features of three different scales are integrated to achieve fine-grained controllability of timbre.
[0055] The timbre generation scheme will be described in detail below with reference to various embodiments.
[0056] Please see Figure 1 , Figure 1 This is a schematic diagram illustrating an application environment for various embodiments of this application. In an exemplary embodiment, the feature fusion device 10 is used to extract fine-grained timbre features, pitch features, and instrument type features of the audio to be generated, and fuses the features at these three scales before inputting them into the generation model 12. The generation model 12 is used to perform noise reduction and diffusion based on the input fused feature data, and outputs a Mel spectrogram of the target timbre. The vocoder 14 is used to process the Mel spectrogram to generate the audio of the target timbre.
[0057] The feature fusion device 10, the generation model 12, and the vocoder 14 can all be applied to the terminal device 1. The terminal device 1 can be a PC (Personal Computer), mobile phone, tablet computer, portable computer, wearable device, etc.
[0058] Example 1
[0059] like Figure 2 The diagram shown is a flowchart of a timbre generation method according to the first embodiment of this application. It should be understood that the flowchart in this method embodiment is not intended to limit the order of execution steps. Depending on the needs, some steps in the flowchart can be added to or removed. The method will now be described using the terminal device as the execution subject.
[0060] The method includes the following steps:
[0061] S200 acquires the required fine-grained timbre features, pitch features, and instrument type features.
[0062] In the embodiments of this application, the first problem to be solved is how to achieve a more objective and unified description of timbre. By establishing a connection between timbre features and digital signal features, the barrier between perceptual description and feature values can be broken down.
[0063] For example, for fine-grained timbre features, seven timbre descriptors can be generated, including Hardness, Depth, Brightness, Roughness, Warmth, Sharpness, and Boominess. Similarly, autoregressive methods based on other timbre perception spaces can be used with neural networks to generate self-selected descriptors. Alternatively, descriptor pairs can be used, describing corresponding timbre features through a pair of antonyms. Examples include: Bright–Dark, Raspy–Mellow, Sharp–Vigorous, Coarse–Pure, and Hoarse–Consonant.
[0064] Once the descriptive terms are fixed, they can be quantified, representing the degree of their descriptive features with a more granular input as a multi-dimensional vector. For example, each timbre descriptive term can be assigned a value between 1 and 100 to represent its intensity. By aggregating these features into fine-grained features, a large amount of timbre features and corresponding timbre description data can be obtained. This allows for the construction of large-scale timbre-description vectors, which can then be used to train neural networks, thus solving the data problem related to timbre description.
[0065] For more details, please refer to the following: Figure 3 This is a schematic diagram illustrating the refinement process for obtaining fine-grained timbre features in step S200 above. Figure 3 In this context, the acquisition of fine-grained timbre features specifically includes:
[0066] S300 provides the user with preset timbre descriptors or descriptor pairs, and a numerical range representing the degree of each descriptor or descriptor pair.
[0067] Each of the descriptive words or pairs of descriptive words corresponds to a timbre characteristic. Examples of descriptive words include Hardness, Depth, Brightness, Roughness, Warmth, Sharpness, and Boominess. Examples of descriptive word pairs include Bright–Dark, Raspy–Mellow, Sharp–Vigorous, Coarse–Pure, and Hoarse–Consonant.
[0068] The numerical range corresponding to the descriptive words is, for example, 1-100, where a smaller value indicates a lower degree of the corresponding timbre feature, and vice versa. For example, for the descriptive word "brightness," a value closer to 1 indicates a darker desired timbre, and a value closer to 100 indicates a brighter desired timbre. The numerical range corresponding to the descriptive word pairs can also be 1-100, where a smaller value indicates a timbre feature closer to the first descriptive word in the pair, and vice versa. For example, for the descriptive word pair "bright-dark," a value closer to 1 indicates a brighter desired timbre, and a value closer to 100 indicates a darker desired timbre.
[0069] In other alternative embodiments, other descriptive words or pairs of descriptive words for timbre features may also be provided, or other forms of numerical ranges may be provided, which will not be elaborated here.
[0070] S302, obtain the numerical value selected by the user for each of the descriptive words or descriptive word pairs, and combine the descriptive words or descriptive word pairs and their corresponding numerical values to obtain the fine-grained timbre feature.
[0071] Once the user selects the descriptive words / descriptive word pairs and their corresponding values for the desired generated timbre, the collection of these descriptive words / descriptive word pairs and their corresponding values is called the fine-grained timbre feature. This fine-grained timbre feature provides a detailed and controllable description of the desired generated timbre, and quantifies each descriptive word / descriptive word pair with a numerical value. This allows for a relatively objective and consistent description of the "degree" of each timbre feature, enabling the differentiation of timbre features at different degrees.
[0072] In addition to the fine-grained timbre features mentioned above, this embodiment still requires instrument type features. Since the fine-grained timbre descriptions are connected to the internal spectral features, they need to be constrained. Adding instrument type information can prevent the generated timbre from deviating from the instrument cluster.
[0073] For more details, please refer to the following: Figure 4 This is a detailed flowchart illustrating the process of obtaining instrument type characteristics in step S200 above. Figure 3 In this context, the acquisition of musical instrument type characteristics specifically includes:
[0074] S400 provides a dataset of musical instrument types.
[0075] S402, Obtain the instrument type label selected by the user based on the instrument type dataset.
[0076] Regarding the instrument type feature, this embodiment provides the instrument types in the dataset as labels to the user, and obtains the instrument type label corresponding to the desired timbre selected by the user.
[0077] S404, Obtain the corresponding instrument type characteristics based on the instrument type label.
[0078] This embodiment can use a method similar to the pitch of the selected instrument type to generate instrument timbre feature data that conforms to the feature fusion strategy, which is the instrument type feature.
[0079] On the other hand, for the same timbre, differences in pitch can also lead to changes in the perceived timbre. From a data perspective, the spectral characteristics within the data of different pitches also differ, and their fine-grained characteristics also vary. To better align with the usage patterns of modern software audio sources, this embodiment uses the fundamental frequency F0 as pitch information, serving as another condition for timbre generation.
[0080] Furthermore, Midi-note (a note-taking software) can be used as a tag to embed monotone labels within the range of human hearing, resulting in pitch features that conform to the feature fusion strategy. In machine learning, embedding refers to the process of mapping high-dimensional data (such as text, images, and audio) to a low-dimensional space.
[0081] Back Figure 2 S202, the fine-grained timbre features, the pitch features, and the instrument type features are fused together.
[0082] In this embodiment, different feature fusion methods can be designed based on the type of generator network.
[0083] For the denoising diffusion generation model, a multi-angle feature fusion method can be used to independently input feature information at different scales into the generation model, using information at different scales to guide the generation of the target timbre. Specifically, for instrument type features and pitch features, the two are concatenated and added to the time step information in the denoising diffusion generation model to form a conditional source. For fine-grained timbre features, they are concatenated with the noise to be denoised (Gaussian noise). Generally, the fine-grained timbre feature information is concatenated at the head of the Gaussian noise to influence the direction of denoising throughout the entire denoising process.
[0084] like Figure 5 The diagram shown is a schematic representation of a feature fusion method based on this embodiment. Figure 5 First, the pitch embedding and the instrument type embedding are concatenated. Then, the concatenated features are fused with the time embedding in the generator model and input into the generator model. Furthermore, the timbre fine-grained features are concatenated with noise and then input into the generator model.
[0085] S204. The fused feature data is input into the trained generative model for denoising and diffusion to obtain the Mel spectrogram of the target timbre.
[0086] In this embodiment, the generative model is the Denoising Diffusion Generation Model (DDPM). The method of generating timbre through denoising diffusion utilizes a trainable Markov chain inverse process to obtain the corresponding Mel spectrogram from Gaussian noise. DDPM comprises two processes: a forward diffusion process and a backward generation process. The forward diffusion process gradually adds Gaussian noise to an image until it becomes random noise, while the backward generation process is the denoising process, gradually removing noise from a random noise source until an image, i.e., the Mel spectrogram of the target timbre, is generated. This is the part that needs to be trained. The training process of the DDPM model can employ existing training methods, which will not be elaborated upon here.
[0087] It is worth noting that in this embodiment, the model input is not simply Gaussian noise, but Gaussian noise with the fine-grained timbre feature information concatenated in the header, and the embedding vector pointing to the time step contains the corresponding pitch and instrument type features. These three types of feature information independently affect the generation of timbre. Unlike the adversarial generation methods commonly used in current technologies, this embodiment is essentially conditional generation.
[0088] Under the premise of using a denoised diffusion generation method, in one implementation, a bilateral denoised diffusion model (BDDM) can be used for inference optimization. This model derives a simplified noise sampling method using the results of intermediate processes, thereby shortening the back-inference process. BDDM generates high-quality samples in far fewer steps than DDPM. BDDM parameterizes the forward and backward processes using scoring and scheduling networks respectively, making it highly efficient and easy to train. Furthermore, any pre-trained DDPM can be further improved by optimizing the inference noise scheduling. In other embodiments, DDPM or DDIM can also be used for back-inference, which will not be elaborated upon here.
[0089] In this embodiment, the ResUnet++ model is used as the denoising network, and the original ResUnet++ model is modified so that it can be applied to the denoising diffusion in the timbre generation process.
[0090] The ResUnet++ model is a semantic segmentation neural network. Generally, the ResUnet++ model consists of an SE (Squeeze and Excitation) module, a Residual Convolution module, an ASPP (Atrous Spatial Pyramid Pooling) module, and an Attention module.
[0091] The SE module recalibrates the feature responses by accurately modeling the interdependencies between channels. The goal of the SE module is to ensure that the network increases its sensitivity to relevant features and suppresses unwanted features. This goal is achieved in two steps: first, by squeezing the global information embedding, which generates channel-specific statistics using global average pooling, thereby compressing each channel; and second, by excitation, which captures the relationships between the modeled channel feature information.
[0092] As network depth increases, training deep neural networks can improve accuracy. However, excessively deep networks can also lead to gradient minimization. The residual module described above makes the network easier to train, and the residual connections within the module help propagate information without degradation.
[0093] The ASPP module successfully resamples features at multiple scales. Within the ASPP module, contextual information is fused with dilated convolutions of varying scales and dilation coefficients from the input feature maps. The Attention module adds the feature maps from the preceding Skip Connection module to the feature map of this layer, effectively performing a feature fusion.
[0094] To conform to the denoising generation process, this embodiment modifies the original ResUnet++ model. First, the timestep embedding vectors are fused in the upsampling and downsampling convolutional layers. Second, the output layer is rewritten, removing the sigmoid function and adjusting the output dimension to maintain the predicted Mel-spectrum between (-1, 1). Finally, a linear layer is added after the last convolutional layer to predict seven-dimensional timbre features; the result is then incorporated into the loss function for optimization.
[0095] The loss function is designed as the sum of the MSELoss (mean squared error loss function) between Mel spectrograms and the Cosine Embedding Loss (cosine similarity loss function) between the seven-dimensional timbre features. The specific calculation formula is as follows:
[0096]
[0097] MSE Loss calculates the similarity between the predicted real timbre Mel spectrum and the generated Mel spectrum. y represents the generated Mel spectrum and the actual Mel spectrum, and N represents the number of predicted spectra and actual spectra.
[0098]
[0099] Li is the EmbeddingLoss, used to calculate the similarity between seven-dimensional timbre features. For a batch of data D(a,b,y) containing N samples, a and b represent two input vectors, and y represents the true class label, belonging to {1,-1}, indicating similarity and dissimilarity, respectively. cos(a i ,b i ) indicates that the vector a is computed. i and b i The cosine of the included angle. Margin is a bias value with a range of [-1, 1], which is used to control the threshold for similarity determination.
[0100] S206, The Mel spectrogram is processed by a vocoder to generate the audio of the target timbre.
[0101] Finally, based on the Mel spectrogram of the target timbre output by the generation model, the speech waveform can be recovered through the processing of the vocoder to obtain the audio of the target timbre.
[0102] like Figure 6 The diagram shown illustrates another form of the timbre generation method proposed in this embodiment. Figure 6 In this process, fine-grained features of timbre, pitch, and instrument type are first extracted through feature engineering. Then, feature fusion is performed, and the fused features are input into the generation model along with the noise to be denoised. The output is the Mel spectrogram of the target timbre. Finally, the waveform of the target timbre audio is obtained through vocoder processing. Figure 6 For details of each step, please refer to the descriptions of the relevant steps above, which will not be repeated here.
[0103] The timbre generation method proposed in this embodiment utilizes the inductive ability of neural networks and the generative ability of generative models to generate timbre. By setting timbre descriptive words / descriptive word pairs and their corresponding numerical ranges, a more objective and unified timbre description is achieved, thereby breaking down the barrier between perceptual descriptions and feature values. During the generation process, features at three different scales—instrument type, pitch, and fine-grained timbre description information—are integrated to achieve fine-grained control over the desired timbre. Unlike the adversarial generation methods commonly used in current technologies, this embodiment employs a conditional generation method in the denoising and diffusion process. Users only need to provide relevant information at three scales to obtain the target timbre inferred by the network, thus enabling personalized timbre customization. The denoising network uses the ResUnet++ model, originally used in other fields, which has been modified to be applicable to the denoising and diffusion process in timbre generation.
[0104] Example 2
[0105] like Figure 7 The diagram shown illustrates the hardware architecture of an electronic device 20 according to a second embodiment of this application. In this embodiment, the electronic device 20 may include, but is not limited to, a memory 21, a processor 22, and a network interface 23, which are interconnected via a system bus. It should be noted that... Figure 7 Only the electronic device 20 with components 21-23 is shown; however, it should be understood that it is not required to implement all the components shown, and more or fewer components may be implemented instead. In this embodiment, the electronic device 20 may be the terminal device.
[0106] The memory 21 includes at least one type of readable storage medium, including flash memory, hard disk, multimedia card, card-type memory (e.g., SD or DX memory), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, magnetic disk, optical disk, etc. In some embodiments, the memory 21 may be an internal storage unit of the electronic device 20, such as the hard disk or memory of the electronic device 20. In other embodiments, the memory 21 may also be an external storage device of the electronic device 20, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, etc., equipped on the electronic device 20. Of course, the memory 21 may include both the internal storage unit and the external storage device of the electronic device 20. In this embodiment, the memory 21 is typically used to store the operating system and various application software installed on the electronic device 20, such as the program code of the tone generation system 60. In addition, the memory 21 can also be used to temporarily store various types of data that have been output or will be output.
[0107] In some embodiments, the processor 22 may be a central processing unit (CPU), controller, microcontroller, microprocessor, or other data processing chip. The processor 22 is typically used to control the overall operation of the electronic device 20. In this embodiment, the processor 22 is used to run program code stored in the memory 21 or process data, for example, to run the tone generation system 60.
[0108] The network interface 23 may include a wireless network interface or a wired network interface, which is typically used to establish a communication connection between the electronic device 20 and other electronic devices.
[0109] Example 3
[0110] like Figure 8 The diagram shown is a modular schematic of a timbre generation system 60 according to a third embodiment of this application. The timbre generation system 60 can be divided into one or more program modules. One or more program modules are stored in a storage medium and executed by one or more processors to complete the embodiments of this application. The program modules referred to in the embodiments of this application are a series of computer program instruction segments capable of performing specific functions. The following description will specifically introduce the functions of each program module in this embodiment.
[0111] In this embodiment, the timbre generation system 60 includes:
[0112] The acquisition module 600 is used to acquire the required fine-grained timbre features, pitch features, and instrument type features.
[0113] In the embodiments of this application, the first problem to be solved is how to achieve a more objective and unified description of timbre. By establishing a connection between timbre features and digital signal features, the barrier between perceptual description and feature values can be broken down.
[0114] The acquisition of fine-grained timbre features specifically includes:
[0115] First, the user is provided with preset timbre descriptors or descriptor pairs, and a numerical range representing the degree of each descriptor or descriptor pair.
[0116] Each of the descriptive words or pairs of descriptive words corresponds to a timbre characteristic. Examples of descriptive words include Hardness, Depth, Brightness, Roughness, Warmth, Sharpness, and Boominess. Examples of descriptive word pairs include Bright–Dark, Raspy–Mellow, Sharp–Vigorous, Coarse–Pure, and Hoarse–Consonant.
[0117] The numerical range corresponding to the descriptive words is, for example, 1-100, where a smaller value indicates a lower degree of the corresponding timbre feature, and vice versa. For example, for the descriptive word "brightness," a value closer to 1 indicates a darker desired timbre, and a value closer to 100 indicates a brighter desired timbre. The numerical range corresponding to the descriptive word pairs can also be 1-100, where a smaller value indicates a timbre feature closer to the first descriptive word in the pair, and vice versa. For example, for the descriptive word pair "bright-dark," a value closer to 1 indicates a brighter desired timbre, and a value closer to 100 indicates a darker desired timbre.
[0118] In other alternative embodiments, other descriptive words or pairs of descriptive words for timbre features may also be provided, or other forms of numerical ranges may be provided, which will not be elaborated here.
[0119] Then, the numerical values selected by the user for each of the descriptive words or pairs of descriptive words are obtained, and the fine-grained timbre features are obtained by combining the descriptive words or pairs of descriptive words and their corresponding numerical values.
[0120] Once the user selects the descriptive words / descriptive word pairs and their corresponding values for the desired generated timbre, the collection of these descriptive words / descriptive word pairs and their corresponding values is called the fine-grained timbre feature. This fine-grained timbre feature provides a detailed and controllable description of the desired generated timbre, and quantifies each descriptive word / descriptive word pair with a numerical value. This allows for a relatively objective and consistent description of the "degree" of each timbre feature, enabling the differentiation of timbre features at different degrees.
[0121] The acquisition of musical instrument type characteristics specifically includes:
[0122] First, a dataset of musical instrument types is provided, and the musical instrument type labels selected by the user based on the dataset are obtained.
[0123] Regarding the instrument type feature, this embodiment provides the instrument types in the dataset as labels to the user, and obtains the instrument type label corresponding to the desired timbre selected by the user.
[0124] Then, the corresponding instrument type characteristics are obtained based on the instrument type label.
[0125] This embodiment can use a method similar to the pitch of the selected instrument type to generate instrument timbre feature data that conforms to the feature fusion strategy, which is the instrument type feature.
[0126] On the other hand, for the same timbre, differences in pitch can lead to variations in timbre perception. From a data perspective, the spectral characteristics within data of different pitches also differ, as do their fine-grained features. To better align with the usage patterns of modern soft audio sources, this embodiment uses the fundamental frequency F0 as pitch information, serving as another condition for timbre generation. Furthermore, Midi-notes can be used as tags to embed the single-tone tags within the range of human hearing, obtaining pitch features that conform to the feature fusion strategy.
[0127] The fusion module 602 is used to fuse the fine-grained timbre features, the pitch features, and the instrument type features.
[0128] For the denoising diffusion generation model, a multi-angle feature fusion method can be used to independently input feature information at different scales into the generation model, using information at different scales to guide the generation of the target timbre. Specifically, for instrument type features and pitch features, the two are concatenated and added to the time step information in the denoising diffusion generation model to form a conditional source. For fine-grained timbre features, they are concatenated with the noise to be denoised (Gaussian noise). Generally, the fine-grained timbre feature information is concatenated at the head of the Gaussian noise to influence the direction of denoising throughout the entire denoising process.
[0129] The denoising diffusion module 604 is used to input the fused feature data into the trained generative model for denoising diffusion to obtain the Mel spectrogram of the target timbre.
[0130] In this embodiment, the generative model is the Denoising Diffusion Generation Model (DDPM). The method of generating timbre through denoising diffusion utilizes a trainable Markov chain inverse process to obtain the corresponding Mel spectrogram from Gaussian noise. DDPM comprises two processes: a forward diffusion process and a backward generation process. The forward diffusion process gradually adds Gaussian noise to an image until it becomes random noise, while the backward generation process is the denoising process, gradually removing noise from a random noise source until an image, i.e., the Mel spectrogram of the target timbre, is generated. This is the part that needs to be trained. The training process of the DDPM model can employ existing training methods, which will not be elaborated upon here.
[0131] It is worth noting that in this embodiment, the model input is not simply Gaussian noise, but Gaussian noise with fine-grained timbre feature information concatenated in the header, and the embedding pointing to the time step contains corresponding pitch and instrument type features. These three types of feature information independently affect the generation of timbre. Unlike the adversarial generation methods commonly used in current technologies, this embodiment is essentially conditional generation.
[0132] Under the premise of using the denoising diffusion generation method, in one implementation, a bilateral denoising diffusion model (BDDM) can be used for inference optimization. The results of the intermediate process are used to derive a simplified noise sampling method to shorten the back inference process. In other embodiments, DDPM or DDIM can also be used for back inference, which will not be elaborated here.
[0133] In this embodiment, the ResUnet++ model is used as the denoising network, and the original ResUnet++ model is modified so that it can be applied to the denoising diffusion in the timbre generation process.
[0134] Processing module 606 is used to process the Mel spectrogram through a vocoder to generate audio of the target timbre.
[0135] Finally, based on the Mel spectrogram of the target timbre output by the generation model, the speech waveform can be recovered through the processing of the vocoder to obtain the audio of the target timbre.
[0136] The timbre generation system proposed in this embodiment utilizes the inductive and generative capabilities of neural networks and generative models to generate timbre. By setting timbre descriptive terms / descriptive term pairs and their corresponding numerical ranges, a more objective and unified timbre description is achieved, thereby breaking down the barrier between perceptual descriptions and feature values. During the generation process, features at three different scales—instrument type, pitch, and fine-grained timbre description information—are integrated to provide fine-grained control over the desired timbre. Unlike the adversarial generation methods commonly used in current technologies, this embodiment employs a conditional generation approach during the denoising and diffusion process. Users only need to provide relevant information at three scales to obtain the target timbre inferred by the network, thus enabling personalized timbre customization. The denoising network uses the ResUnet++ model, originally used in other fields, which has been modified to be applicable to denoising and diffusion in the timbre generation process.
[0137] Example 4
[0138] This application also provides another embodiment, namely, a computer-readable storage medium storing a timbre generation program, which can be executed by at least one processor to cause the at least one processor to perform the steps of the timbre generation method as described above.
[0139] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.
[0140] The sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.
[0141] Obviously, those skilled in the art should understand that the modules or steps of the embodiments of this application described above can be implemented using general-purpose computing devices. They can be centralized on a single computing device or distributed across a network of multiple computing devices. Optionally, they can be implemented using computer-executable program code, thereby storing them in a storage device for execution by a computing device. In some cases, the steps shown or described can be performed in a different order than those presented here, or they can be fabricated as separate integrated circuit modules, or multiple modules or steps can be fabricated as a single integrated circuit module. Thus, the embodiments of this application are not limited to any particular combination of hardware and software.
[0142] The above are merely preferred embodiments of the present application and do not limit the patent scope of the present application. Any equivalent structural or procedural transformations made using the description and drawings of the present application, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of the present application.
Claims
1. A timbre generation method characterized by, The method includes: Obtain the required fine-grained timbre features, pitch features, and instrument type features; The fine-grained timbre features, the pitch features, and the instrument type features are fused together. The fused feature data is input into the trained generative model for denoising and diffusion to obtain the Mel spectrogram of the target timbre. The Mel spectrogram is processed by a vocoder to generate the audio of the target timbre; The feature fusion of the fine-grained timbre features, the pitch features, and the instrument type features includes: The pitch feature and the instrument type feature are concatenated, and the concatenated feature is fused with the time step information in the generation model; The fine-grained timbre features are spliced together with the noise to be denoised; The generation model includes a denoising diffusion generation model.
2. The timbre generation method according to claim 1, characterized by, The acquisition of fine-grained timbre features includes: Provide users with preset timbre descriptors or descriptor pairs, and a numerical range representing the degree of each descriptor or descriptor pair, wherein each descriptor or descriptor pair corresponds to a timbre feature; Obtain the numerical value selected by the user for each of the descriptive words or descriptive word pairs, and combine the descriptive words or descriptive word pairs and their corresponding numerical values to obtain the fine-grained timbre feature.
3. The timbre generation method according to claim 1, characterized by, The characteristics of the musical instrument types obtained include: Provides a dataset of musical instrument types; Obtain the instrument category label selected by the user based on the instrument category dataset; The corresponding instrument type characteristics are obtained based on the instrument type label.
4. The timbre generation method according to claim 2, characterized by, The provision of preset timbre descriptive words or pairs of descriptive words to the user, and the numerical range of each of the descriptive words or pairs of descriptive words, include: Provide users with timbre descriptions including hardness, depth, brightness, roughness, warmth, sharpness, and low-frequency resonance, and set the numerical range of each description to 1 to 100.
5. The timbre generation method according to claim 1, characterized in that, The step of inputting the fused feature data into the trained generative model for denoising and diffusion to obtain the Mel spectrogram of the target timbre includes: The denoising diffusion generation model is input with Gaussian noise whose head is concatenated with the fine-grained timbre feature information, and the embedding vector pointing to the time step contains the pitch feature and the instrument type feature information, to obtain the Mel spectrum of the target timbre output by the denoising diffusion generation model.
6. The timbre generation method according to claim 1 or 5, characterized in that, The generative model uses a bilateral denoising diffusion model for inference optimization, shortening the reverse inference process.
7. The timbre generation method according to claim 1 or 5, characterized in that, The generative model uses an improved ResUnet++ model as the denoising network.
8. A timbre generation system, characterized in that, The system includes: The acquisition module is used to acquire the required fine-grained timbre features, pitch features, and instrument type features; The fusion module is used to fuse the fine-grained timbre features, the pitch features, and the instrument type features. The denoising and diffusion module is used to input the fused feature data into the trained generative model for denoising and diffusion, and to obtain the Mel spectrogram of the target timbre. The processing module is used to process the Mel spectrogram through a vocoder to generate the audio of the target timbre; The feature fusion of the fine-grained timbre features, the pitch features, and the instrument type features includes: The pitch feature and the instrument type feature are concatenated, and the concatenated feature is fused with the time step information in the generation model; The fine-grained timbre features are spliced together with the noise to be denoised; The generation model includes a denoising diffusion generation model.
9. An electronic device, characterized in that, The electronic device includes: a memory, a processor, and a timbre generation program stored in the memory and executable on the processor, wherein the timbre generation program, when executed by the processor, implements the timbre generation method as described in any one of claims 1 to 7.
10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a timbre generation program, which, when executed by a processor, implements the timbre generation method as described in any one of claims 1 to 7.
11. A computer program product comprising computer instructions, characterized in that, When executed by a processor, the computer instructions implement the steps of the timbre generation method as described in any one of claims 1 to 7.