A Timbre Conversion Method and System Based on Machine Learning Algorithms

Abstract

This application provides a timbre conversion method and system based on machine learning algorithms. First, it acquires a source audio signal and a target timbre description. The source audio signal is decomposed into multiple audio frames, and spectral features are extracted. The target timbre description is converted into a multi-dimensional timbre feature vector. Then, feature fusion is performed through a timbre fusion network to obtain the converted spectral features. Finally, the target audio signal is generated through inverse time-frequency transformation. Furthermore, the application includes steps such as adjusting the dynamic range of the converted spectral features and post-processing the target audio signal to improve the quality and effect of timbre conversion and solve timbre conversion problems under various complex conditions.

Description

Technical Field

[0001] This application relates to the field of machine learning technology, and more specifically, to a timbre conversion method and system based on machine learning algorithms. Background Technology

[0002] In the field of audio processing, timbre conversion is a crucial technology. Different timbres can convey different emotions, information, and styles. For example, in speech synthesis, users may want to convert a bland voice into an emotional, magnetic, or softer timbre; in music production, it is often necessary to convert the timbre of one instrument into that of another to achieve unique musical effects. Traditional timbre conversion methods often have many limitations, such as producing an unnatural timbre, poor preservation of the original audio's speech content, or unsatisfactory results when processing complex timbre attributes. Summary of the Invention

[0003] In view of this, the purpose of this application is to provide a timbre conversion method and system based on machine learning algorithms.

[0004] According to a first aspect of this application, a timbre conversion method based on a machine learning algorithm is provided, the method comprising:

[0005] Acquire a source audio signal and a target timbre description, wherein the source audio signal contains the original timbre features to be converted, and the target timbre description is used to define the timbre attributes of the desired output;

[0006] The source audio signal is decomposed into multiple audio frames, and time-frequency transformation processing is performed on each audio frame to extract the spectral features of the source audio signal;

[0007] The target timbre description is converted into a multi-dimensional timbre feature vector, which contains quantization parameters corresponding to the timbre attribute.

[0008] The spectral features and the multidimensional timbre feature vector are input into the timbre fusion network for feature fusion to generate transformed spectral features containing the target timbre attributes;

[0009] The converted spectral features are subjected to inverse time-frequency transformation to generate a target audio signal, the timbre attributes of which are consistent with the desired timbre attributes in the target timbre description.

[0010] In one possible implementation of the first aspect, the step of decomposing the source audio signal into multiple audio frames and performing time-frequency transformation processing on each audio frame to extract the spectral features of the source audio signal includes:

[0011] The source audio signal is segmented into frames to obtain multiple audio frames of fixed duration. Each audio frame is processed by a windowing function to reduce spectral leakage.

[0012] Perform a Fourier transform on each windowed audio frame to obtain the corresponding amplitude spectrum and phase spectrum;

[0013] Mel frequency cepstral coefficients are extracted from the amplitude spectrum as the spectral features, and the phase spectrum is retained as a reconstruction parameter.

[0014] In one possible implementation of the first aspect, the target timbre description includes a reference audio segment or a text description entered by the user;

[0015] When the target timbre is described as a reference audio segment, the reference audio segment is subjected to fundamental frequency extraction and formant analysis to generate the multidimensional timbre feature vector;

[0016] When the target timbre description is a text description, a pre-trained timbre attribute parsing model is invoked to map the text description into quantization parameters in the multi-dimensional timbre feature vector.

[0017] In one possible implementation of the first aspect, the timbre fusion network includes an encoder and a decoder, the encoder consisting of multiple convolutional layers for compressing the spectral features into latent spatial features;

[0018] The decoder consists of multiple deconvolutional layers, which are used to reconstruct the converted spectral features by concatenating the latent spatial features and the multidimensional timbre feature vectors through channels;

[0019] The encoder and the decoder transmit low-frequency information through skip connections to preserve the speech content of the source audio signal.

[0020] In one possible implementation of the first aspect, the training process of the timbre fusion network includes:

[0021] Collect multiple sets of training audio data with different timbres, and label each set of training audio data with corresponding timbre attribute tags;

[0022] The spectral features of the training audio are input into the encoder to obtain latent features, and the labeled timbre attribute tags are converted into conditional vectors.

[0023] The latent features and the conditional vector are input into the decoder to generate reconstructed spectral features;

[0024] Calculate the mean square error loss between the reconstructed spectral features and the target spectral features, and jointly optimize the network parameters of the encoder and the decoder.

[0025] In one possible implementation of the first aspect, the method further includes the step of dynamically adjusting the converted spectral features:

[0026] Based on the fundamental frequency trajectory and energy distribution of the source audio signal, the ratio of harmonic components and noise components in the converted spectral characteristics is adjusted;

[0027] The adjusted spectral characteristics are combined with the retained phase spectrum to generate an intermediate signal that satisfies time-domain continuity.

[0028] Linear prediction analysis is performed on the intermediate signal to correct the spectral envelope to match the formant bandwidth parameter in the target timbre description.

[0029] In one possible implementation of the first aspect, the dynamic range adjustment is achieved via a differentiable signal processing module, the differentiable signal processing module comprising:

[0030] The harmonic enhancement unit is used to enhance high-frequency harmonic energy based on the brightness parameter in the multidimensional timbre feature vector.

[0031] A noise suppression unit is used to reduce the amplitude of non-periodic noise based on the smoothness parameter in the multidimensional timbre feature vector;

[0032] The outputs of the harmonic enhancement unit and the noise suppression unit are fused by a weighted summation method, with the weights dynamically controlled by the emotion category parameter in the timbre attribute.

[0033] In one possible implementation of the first aspect, the method further includes a step of post-processing the target audio signal:

[0034] Extract the original rhythmic features and intonation contours of the source audio signal, and time-align the original rhythmic features with the spectrum of the target audio signal;

[0035] The fundamental frequency curve of the target audio signal is adjusted according to the intonation profile to maintain the same speech rate and stress pattern as the source audio signal.

[0036] The adjusted baseband curve is input into the vocoder to generate the final target audio signal.

[0037] In one possible implementation of the first aspect, when the user-input text description contains multiple conflicting timbre attributes, the following processing is performed:

[0038] The weight coefficient of each timbre attribute is calculated through an attention mechanism, and the weight coefficient is determined based on the compatibility between the original timbre features of the source audio signal and the target timbre attribute.

[0039] The weighted timbre attribute parameters are merged into a unified multidimensional timbre feature vector, and a corresponding conflict resolution log is generated for user confirmation.

[0040] According to a second aspect of this application, a timbre conversion system based on a machine learning algorithm is provided. The timbre conversion system based on a machine learning algorithm includes a machine-readable storage medium and a processor. The machine-readable storage medium stores machine-executable instructions. When the processor executes the machine-executable instructions, the timbre conversion system based on a machine learning algorithm implements the aforementioned timbre conversion method based on a machine learning algorithm.

[0041] According to a third aspect of this application, a computer-readable storage medium is provided, wherein computer-executable instructions are stored therein, and when the computer-executable instructions are executed, the aforementioned timbre conversion method based on machine learning algorithms is implemented.

[0042] Based on any of the above aspects, the technical effect of this application is as follows:

[0043] First, the source audio signal and target timbre description are acquired. The source audio signal is decomposed into multiple audio frames and its spectral features are extracted. The target timbre description is converted into a multi-dimensional timbre feature vector. Then, the features are fused through a timbre fusion network to obtain the converted spectral features. Finally, the target audio signal is generated through inverse time-frequency transformation. Furthermore, the process involves adjusting the dynamic range of the converted spectral features and post-processing the target audio signal to improve the quality and effect of timbre conversion and solve timbre conversion problems under various complex conditions. Attached Figure Description

[0044] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0045] Figure 1 The flowchart of the timbre conversion method based on machine learning algorithm provided in this application embodiment is shown in the figure.

[0046] Figure 2 A schematic diagram of the component structure of the timbre conversion system based on machine learning algorithms provided in an embodiment of this application is shown. Detailed Implementation

[0047] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. It should be understood that the accompanying drawings in this application are for illustrative and descriptive purposes only and are not intended to limit the scope of protection of this application. Furthermore, it should be understood that the schematic drawings are not drawn to scale. The flowcharts used in this application illustrate operations implemented according to some embodiments of this application. It should be understood that the operations in the flowcharts may not be implemented in sequence, and steps without logical contextual relationships may be reversed or implemented simultaneously. In addition, those skilled in the art, guided by the content of this application, can add multiple other operations to the flowcharts, or delete multiple operations from the flowcharts.

[0048] Figure 1 This document illustrates a flowchart of a timbre conversion method and system based on a machine learning algorithm provided in an embodiment of this application. It should be understood that in other embodiments, the order of some steps in the timbre conversion method based on the machine learning algorithm of this embodiment can be shared according to actual needs, or some steps can be omitted or maintained. The detailed steps of the timbre conversion method based on the machine learning algorithm include:

[0049] Step S110: Obtain the source audio signal and the target timbre description, wherein the source audio signal contains the original timbre features to be converted, and the target timbre description is used to define the timbre attributes of the desired output.

[0050] In a real-world audio processing scenario, suppose a speech synthesis project is underway. The source audio signal could be a recording of a male announcer reading a news report. This voice possesses the announcer's unique vocal characteristics, such as a relatively low, steady tone, clear pronunciation, and possibly a slight nasal quality. The target vocal description, on the other hand, is the desired vocal attribute determined by the project requirements. For example, the target vocal description might be to transform the male announcer's voice into the vocal quality of a young woman, which might have a higher pitch, a crisper tone, and a softer sound. The target vocal description can also be defined in various ways. If there is a reference audio clip of a young woman reading the same news report, then that reference audio clip can serve as the target vocal description; or it could be a text description directly entered by the user, such as "young woman's vocal quality, higher pitch, crisp and soft sound."

[0051] Step S120: Decompose the source audio signal into multiple audio frames, and perform time-frequency transformation processing on each audio frame to extract the spectral features of the source audio signal.

[0052] For the source audio signal of the male announcer mentioned above, frame segmentation is performed first. The entire audio is divided into multiple audio frames of fixed duration according to certain rules. For example, if the duration of each audio frame is set to 20 milliseconds, the entire source audio signal is divided into a series of 20-millisecond segments. After framing, in order to reduce spectral leakage, a windowing function needs to be applied to each audio frame. Here, the Hanning window function can be used, which can effectively reduce spectral leakage.

[0053] Next, a Fourier transform is performed on each windowed audio frame. The Fourier transform is a powerful tool for converting time-domain signals to frequency-domain signals. Through the Fourier transform, the amplitude spectrum and phase spectrum corresponding to each audio frame can be obtained. The amplitude spectrum reflects the energy distribution of the signal at different frequencies, while the phase spectrum contains the phase information of the signal. Mel-frequency cepstral coefficients are extracted from the amplitude spectrum as spectral features. Mel-frequency cepstral coefficients are a widely used feature in speech processing, as they can well represent the spectral characteristics of speech. Simultaneously, the phase spectrum is retained as a reconstruction parameter because it is crucial for restoring the time-domain characteristics of the audio signal during the subsequent inverse transform.

[0054] Step S130: Convert the target timbre description into a multi-dimensional timbre feature vector, wherein the multi-dimensional timbre feature vector contains quantization parameters corresponding to the timbre attributes.

[0055] If the target timbre description is a reference audio segment, such as the aforementioned reference audio segment of a young woman reading a news report, the first step is to extract the fundamental frequency and analyze the formants. The fundamental frequency is the most basic frequency component in speech, determining the pitch. A specific algorithm, such as the autocorrelation function method, is used to extract the fundamental frequency of the reference audio segment. Formants are frequency regions in the sound spectrum where energy is relatively concentrated; they reflect the shape and characteristics of the vocal tract. Methods such as linear predictive coding (LPC) are used to analyze the formants. Based on the fundamental frequency and formant information, a multidimensional timbre feature vector is generated. The quantization parameters in this multidimensional timbre feature vector accurately describe various attributes of the target timbre, such as the specific pitch value, the frequency of the formants, and the bandwidth.

[0056] If the target timbre description is a user-inputted text description, such as "young female timbre, high pitch, clear and mellow voice," a pre-trained timbre attribute parsing model is invoked. This model is trained on a large number of timbre samples and corresponding text descriptions. The user-inputted text description is fed into this model, and the model, based on its learned knowledge, maps the timbre attributes in the text to quantization parameters in a multi-dimensional timbre feature vector. For example, the attribute "high pitch" might be mapped to a quantization parameter with a large fundamental frequency value, and "clear and mellow voice" might be mapped to quantization parameters related to spectral smoothness, harmonic structure, etc.

[0057] Step S140: Input the spectral features and the multidimensional timbre feature vector into the timbre fusion network for feature fusion to generate a transformed spectral feature containing the target timbre attributes.

[0058] The timbre fusion network consists of an encoder and a decoder. The encoder comprises multiple convolutional layers, for example, three. The spectral features of the previously extracted source audio signal are fed into this encoder. The first convolutional layer might use a 3×3 kernel with a stride of 1 to perform initial compression and feature extraction of the spectral features. As the signal passes through multiple convolutional layers, the spectral features are gradually compressed into latent spatial features.

[0059] The decoder consists of multiple deconvolutional layers, let's assume three. During this process, the latent spatial features and the multidimensional timbre feature vector are concatenated channel by channel. For example, if the latent spatial features have 16 channels and the multidimensional timbre feature vector has 8 channels, the resulting new feature has 24 channels. Then, these deconvolutional layers reconstruct this concatenated feature into the transformed spectral features. In this process, the encoder and decoder pass low-frequency information through skip connections to preserve the speech content of the source audio signal. For instance, the low-frequency information output from the first convolutional layer of the encoder is directly passed to the corresponding layer of the decoder through skip connections. This ensures that the speech content in the original audio is not lost during timbre conversion; for example, word pronunciation and rhythm are well preserved.

[0060] Step S150: Perform inverse time-frequency transformation on the converted spectral features to generate a target audio signal, wherein the timbre attributes of the target audio signal are consistent with the desired timbre attributes in the target timbre description.

[0061] Having obtained the converted spectral characteristics through the preceding steps, we now need to perform an inverse time-frequency transform. The inverse time-frequency transform is the reverse process of the previous time-frequency transform; it converts the frequency domain signal back to the time domain signal. In this process, using the previously preserved phase spectrum, the converted spectral characteristics are combined with the phase spectrum. Through operations such as the inverse Fourier transform, the target audio signal is generated. The timbre attributes of this target audio signal are consistent with the desired timbre attributes in the target timbre description. For example, if the target timbre description is the timbre of a young woman, then the generated target audio signal should possess the characteristics of a young woman's timbre, such as a higher pitch and a clear, mellow tone.

[0062] Step S160: The step of dynamically adjusting the range of the converted spectral features.

[0063] Based on the fundamental frequency trajectory and energy distribution of the source audio signal, the ratio of harmonic and noise components in the converted spectral characteristics is adjusted. In the example mentioned earlier of a male announcer converting to a young female voice, the male announcer's source audio signal has a specific fundamental frequency trajectory and energy distribution. The fundamental frequency trajectory reflects the change in pitch over time, while the energy distribution represents the energy level at different frequencies. When converting to a young female voice, it may be necessary to adjust the ratio of harmonic and noise components according to the characteristics of a young female voice. For example, a young female voice may have relatively richer harmonic components and less noise components.

[0064] The adjusted spectral characteristics are combined with the preserved phase spectrum to generate an intermediate signal that satisfies time-domain continuity. This step is to ensure that the generated signal is continuous in the time domain and does not exhibit sudden jumps or unnatural phenomena.

[0065] Linear predictive analysis is performed on the intermediate signal to correct the spectral envelope and match the formant bandwidth parameter in the target timbre description. Formant bandwidth is an important characteristic of timbre; for a young female timbre, the formant bandwidth may differ from that of a male announcer. Through linear predictive analysis, the spectral envelope can be corrected so that the formant bandwidth of the generated target audio signal matches the parameter in the target timbre description, thereby more accurately converting the timbre into the desired young female timbre.

[0066] This dynamic range adjustment is achieved through a differentiable signal processing module, which includes a harmonic enhancement unit and a noise suppression unit. The harmonic enhancement unit amplifies high-frequency harmonic energy based on the brightness parameter in the multidimensional timbre feature vector. For example, if the target timbre description requires a brighter sound, the brightness parameter will be higher, and the harmonic enhancement unit will correspondingly amplify the high-frequency harmonic energy, making the sound brighter. The noise suppression unit reduces the amplitude of non-periodic noise based on the smoothness parameter in the multidimensional timbre feature vector. If the target timbre requires a smoother sound, the smoothness parameter will prompt the noise suppression unit to reduce the noise amplitude, making the sound purer. The outputs of the harmonic enhancement and noise suppression units are fused using a weighted summation method, with the weights dynamically controlled by the emotion category parameter in the timbre attribute. For example, if the target timbre is a cheerful young female timbre, the emotion category parameter might cause the fusion to favor enhancing harmonic energy to convey a cheerful feeling.

[0067] Step S170: A step of post-processing the target audio signal.

[0068] The original rhythmic features and intonation profiles of the source audio signal are extracted, and the original rhythmic features are time-aligned with the spectrum of the target audio signal. For the source audio signal of a male announcer, his reading has certain rhythmic features, such as the pause time between each word and the distribution of stress in sentences, while the intonation profile reflects the fluctuations in pitch. After converting the male announcer's voice into a young female voice, these original rhythmic features are time-aligned with the spectrum of the target audio signal. This is like applying the original rhythmic framework to the new timbre, so that the new timbre is rhythmically consistent with the original sound.

[0069] The fundamental frequency curve of the target audio signal is adjusted based on the intonation profile to maintain the same speech rate and stress pattern as the source audio signal. For example, when a male announcer reads a news report, certain important words are stressed, and the sentence speed follows a certain pattern. In the target audio signal with the converted young female voice, the fundamental frequency curve is adjusted according to the previously extracted intonation profile to ensure that the speech rate and stress pattern are the same as the source audio signal. The purpose of this is to ensure that the converted voice does not create ambiguity in its expression and sounds more natural and fluent.

[0070] The adjusted fundamental frequency curve is input into a vocoder to generate the final target audio signal. A vocoder is a device that converts parameters such as the fundamental frequency curve into an audible audio signal. Through the processing of the vocoder, the quality of the target audio signal can be further optimized, so that the final generated target audio signal not only has the desired young female voice timbre, but also maintains important characteristics such as rhythm, speech rate, and stress patterns similar to the source audio signal.

[0071] Step S180: Processing steps when the text description entered by the user contains multiple conflicting timbre attributes.

[0072] Suppose a user inputs the text description "both a deep tone and a crisp timbre," which is a description containing conflicting timbre attributes. An attention mechanism calculates a weight coefficient for each timbre attribute, determined based on the compatibility between the original timbre features of the source audio signal and the target timbre attribute. In the case of a male announcer as the source audio signal, a deep tone has some compatibility with the original timbre because male announcers naturally have relatively deep tones; while a crisp timbre has lower compatibility. The attention mechanism calculates the weight coefficient for each timbre attribute based on this compatibility.

[0073] The weighted timbre attribute parameters are merged into a unified multi-dimensional timbre feature vector, and a corresponding conflict resolution log is generated for user confirmation. For example, based on the calculated weight coefficients, a deep pitch attribute might be assigned a higher weight, while a crisp tone attribute might be assigned a lower weight. These weighted timbre attribute parameters are then merged into a unified multi-dimensional timbre feature vector. Simultaneously, a conflict resolution log is generated, detailing the original requirements for each timbre attribute, the calculated weight coefficients, and the final merging process, for user review and confirmation to ensure the timbre conversion result meets user expectations.

[0074] Figure 2 The illustrated timbre conversion system 100 based on machine learning algorithms includes a processor 1001 and a memory 1003. The processor 1001 and memory 1003 are connected, for example, via a bus 1002. Optionally, the timbre conversion system 100 may further include a transceiver 1004, which can be used for data interaction between the server and other servers, such as sending and / or receiving data. It should be noted that in actual scheduling, the transceiver 1004 is not limited to one unit, and the structure of this timbre conversion system 100 based on machine learning algorithms does not constitute a limitation on the embodiments of this application.

[0075] Processor 1001 may be a CPU (Central Processing Unit), a general-purpose processor, a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. It can implement or execute the various exemplary logic blocks, modules, and circuits described in conjunction with the disclosure of this application. Processor 1001 may also be a combination that implements computing functions, such as including one or more microprocessor combinations, a combination of a DSP and a microprocessor, etc.

[0076] Bus 1002 may include a pathway for transmitting information between the aforementioned components. Bus 1002 may be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus, etc. Bus 1002 may be divided into address bus, data bus, control bus, etc. For ease of representation, Figure 2 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0077] The memory 1003 may be ROM (Read Only Memory) or other types of static storage devices capable of storing static information and instructions, RAM (Random Access Memory) or other types of dynamic storage devices capable of storing information and instructions, or EEPROM (Electrically Erasable Programmable Read Only Memory), CD-ROM (Compact Disc Read Only Memory) or other optical disc storage, optical disc storage (including compressed optical discs, laser discs, optical discs, digital universal optical discs, Blu-ray discs, etc.), magnetic disk storage media, other magnetic storage devices, or any other medium capable of carrying or storing program code and capable of being read by a computer, without limitation herein.

[0078] The memory 1003 is used to store program code for executing the embodiments of this application, and its execution is controlled by the processor 1001. The processor 1001 is used to execute the program code stored in the memory 1003 to implement the steps shown in the foregoing method embodiments.

[0079] This application provides a computer-readable storage medium storing program code, which, when executed by a processor, can implement the steps and corresponding content of the aforementioned method embodiments.

[0080] It should be understood that although directed lines represent various operation steps in the flowcharts of the embodiments of this application, the order in which these steps are implemented is not limited to the order covered by the directed lines. Unless explicitly stated herein, in some implementation scenarios of the embodiments of this application, the implementation steps in each flowchart may be executed in a different order based on requirements. Furthermore, some or all of the steps in each flowchart may include multiple sub-steps or multiple stages depending on the actual implementation scenario. Some or all of these sub-steps or stages may be executed in the same stage, or each sub-step or stage may be executed in different stages. In scenarios with different execution stages, the execution order of these sub-steps or stages can be flexibly configured based on requirements, and the embodiments of this application do not limit this.

[0081] The above description is only an optional implementation method for some implementation scenarios of this application. It should be noted that for those skilled in the art, other similar implementation methods based on the technical concept of this application, without departing from the technical concept of this application, also fall within the protection scope of the embodiments of this application.

Claims

1. A timbre conversion method based on machine learning algorithms, characterized in that, The method includes: Acquire a source audio signal and a target timbre description, wherein the source audio signal contains the original timbre features to be converted, and the target timbre description is used to define the timbre attributes of the desired output; The source audio signal is decomposed into multiple audio frames, and time-frequency transformation processing is performed on each audio frame to extract the spectral features of the source audio signal; The target timbre description is converted into a multi-dimensional timbre feature vector, which contains quantization parameters corresponding to the timbre attribute. The spectral features and the multidimensional timbre feature vector are input into the timbre fusion network for feature fusion to generate transformed spectral features containing the target timbre attributes; The converted spectral features are subjected to inverse time-frequency transformation to generate a target audio signal, wherein the timbre attributes of the target audio signal are consistent with the desired timbre attributes in the target timbre description; The target timbre description includes a reference audio segment or a text description input by the user; when the target timbre description is a reference audio segment, the fundamental frequency is extracted and formant analysis is performed on the reference audio segment to generate the multidimensional timbre feature vector; when the target timbre description is a text description, a pre-trained timbre attribute parsing model is invoked to map the text description into quantization parameters in the multidimensional timbre feature vector; The timbre fusion network includes an encoder and a decoder. The encoder consists of multiple convolutional layers, used to compress the spectral features into latent spatial features. The decoder consists of multiple deconvolutional layers, used to reconstruct the transformed spectral features by concatenating the latent spatial features with the multidimensional timbre feature vector through channels. The encoder and the decoder transmit low-frequency information through skip connections to preserve the speech content of the source audio signal. The training process of the timbre fusion network includes: collecting multiple sets of training audio data with different timbres and labeling each set of training audio data with corresponding timbre attribute tags; inputting the spectral features of the training audio into the encoder to obtain latent features, and converting the labeled timbre attribute tags into conditional vectors; inputting the latent features and the conditional vectors into the decoder to generate reconstructed spectral features; calculating the mean squared error loss between the reconstructed spectral features and the target spectral features, and jointly optimizing the network parameters of the encoder and the decoder.

2. The timbre conversion method based on machine learning algorithm according to claim 1, characterized in that, The step of decomposing the source audio signal into multiple audio frames and performing time-frequency transform processing on each audio frame to extract the spectral features of the source audio signal includes: performing frame segmentation processing on the source audio signal to obtain multiple audio frames of fixed duration, and processing each audio frame through a windowing function to reduce spectral leakage; performing a Fourier transform on each windowed audio frame to obtain the corresponding amplitude spectrum and phase spectrum; extracting Mel-frequency cepstral coefficients from the amplitude spectrum as the spectral features, and retaining the phase spectrum as a reconstruction parameter.

3. The timbre conversion method based on machine learning algorithm according to claim 2, characterized in that, The method further includes a step of dynamically adjusting the converted spectral features: adjusting the ratio of harmonic components and noise components of the converted spectral features according to the fundamental frequency trajectory and energy distribution of the source audio signal; combining the adjusted spectral features with the retained phase spectrum to generate an intermediate signal that satisfies time-domain continuity; performing linear prediction analysis on the intermediate signal to correct the spectral envelope to match the formant bandwidth parameter in the target timbre description.

4. The timbre conversion method based on machine learning algorithm according to claim 3, characterized in that, The dynamic range adjustment is achieved through a differentiable signal processing module, which includes: a harmonic enhancement unit for enhancing high-frequency harmonic energy based on the brightness parameter in the multidimensional timbre feature vector; and a noise suppression unit for reducing the amplitude of non-periodic noise based on the smoothness parameter in the multidimensional timbre feature vector. The outputs of the harmonic enhancement unit and the noise suppression unit are fused by a weighted summation method, with the weights dynamically controlled by the emotion category parameter in the timbre attribute.

5. The timbre conversion method based on machine learning algorithm according to claim 1, characterized in that, The method further includes a post-processing step for the target audio signal: extracting the original rhythmic features and intonation profile of the source audio signal, and aligning the original rhythmic features with the spectrum of the target audio signal in time; adjusting the fundamental frequency curve of the target audio signal according to the intonation profile to maintain the same speech rate and stress pattern as the source audio signal; and inputting the adjusted fundamental frequency curve into a vocoder to generate the final target audio signal.

6. The timbre conversion method based on machine learning algorithm according to claim 1, characterized in that, When the text description input by the user contains multiple conflicting timbre attributes, the following processing is performed: A weight coefficient for each timbre attribute is calculated using an attention mechanism, the weight coefficient being determined based on the compatibility between the original timbre features of the source audio signal and the target timbre attribute; The weighted timbre attribute parameters are merged into a unified multidimensional timbre feature vector, and a corresponding conflict resolution log is generated for user confirmation.

7. A timbre conversion system based on machine learning algorithms, characterized in that, The device includes a processor and a computer-readable storage medium storing machine-executable instructions that, when executed by the processor, implement the timbre conversion method based on a machine learning algorithm as described in any one of claims 1-6.

Images