Tone conversion processing method, device, electronic device and storage medium

Abstract

The present disclosure relates to a timbre conversion processing method, device, electronic device and storage medium. The method includes: obtaining original audio and extracting timbre conversion information of the original audio; synthesizing reference audio with a target timbre according to the timbre conversion information, wherein the reference audio at least includes harmonic audio, and the harmonic audio is an audio that is consistent with the audio content of the original audio and only includes harmonics; obtaining the sinusoidal component of the original audio, and interpolating the harmonic audio and the sinusoidal component based on a preset interpolation coefficient through a spectral feature interpolation method to obtain a target audio after timbre conversion, wherein the timbre of the target audio is between the timbre of the original audio and the target timbre. In this way, after synthesizing the reference audio with the target timbre, a controllable timbre conversion scheme is realized in combination with the time-frequency domain feature interpolation technology based on the study of auditory perception characteristics, thereby improving the effectiveness and flexibility of timbre conversion.

Description

Technical Field

[0001] The present disclosure relates to the technical field of audio processing, and in particular to a timbre conversion processing method, device, electronic device and storage medium. Background Art

[0002] Controllable instrument timbre conversion technology is used to gradually convert the timbre of one instrument to another. The sound processed by this technology has any transitional timbre between the original timbre and the target timbre. While converting the timbre, the musical content such as pitch, duration and intensity remain unchanged. This technology can be applied to music production, sound effect generation and other fields, providing new design ideas for artistic creation such as electronic music timbre design, arrangement and recording post-production.

[0003] In the related art, one method is to indirectly cause harmonic changes by adjusting the time-frequency spectrum envelope of the sound, convolution of the pre-prepared impulse response, compression or extension, pitch change or phase shift, thereby affecting the timbre. However, the main factor affecting the timbre is the harmonic distribution of the sound signal, not the time-frequency spectrum envelope curve, and the above method cannot control the timbre attributes of the sound alone, so that changing the timbre will inevitably cause changes in other sound attributes. Another method is to use an end-to-end machine learning model to process sound generation and conversion to achieve the purpose of timbre conversion. However, the timbre conversion of this method is uncontrollable, and can only convert the timbre of the original sound to the timbre of the target sound, and cannot produce a sound with a transitional timbre between the two timbres.

[0004] It can be seen that the above-mentioned timbre adjustment method is difficult to perform individual conversion control on timbre, and it is difficult to produce a sound with a transitional timbre between two timbres. Summary of the invention

[0005] The present disclosure provides a timbre conversion processing method, device and system to at least solve the problem in the related art that it is difficult to perform individual conversion control on timbre and it is difficult to produce a sound with a transitional timbre between two timbres. The technical solution of the present disclosure is as follows:

[0006] According to a first aspect of an embodiment of the present disclosure, a method for tone conversion is provided, comprising:

[0007] Acquire original audio, and extract timbre conversion information of the original audio;

[0008] Synthesize a reference audio with a target timbre according to the timbre conversion information, wherein the reference audio at least includes harmonic audio, and the harmonic audio is an audio that is consistent with the audio content of the original audio and only includes harmonics;

[0009] The sinusoidal component of the original audio is obtained, and based on a preset interpolation coefficient, the harmonic audio and the sinusoidal component are interpolated by a spectral feature interpolation method to obtain a target audio after timbre conversion, wherein the timbre of the target audio is between the timbre of the original audio and the target timbre.

[0010] Optionally, the step of synthesizing reference audio with a target timbre according to the timbre conversion information includes:

[0011] synthesizing reference audio with a target timbre through a decoder component according to the timbre conversion information;

[0012] The decoder component is obtained by training the autoencoder model to be trained, and the autoencoder model to be trained includes at least one decoder component to be trained.

[0013] Optionally, before the step of synthesizing reference audio with target timbre through a decoder component according to the timbre conversion information, the method further includes:

[0014] Acquire at least one sample audio and a standard audio corresponding to the sample audio, wherein the standard audio has a target timbre and is consistent with the audio content of the sample audio;

[0015] Inputting the sample audio into the autoencoder model to be trained to obtain predicted audio;

[0016] A loss value between the predicted audio and the standard audio is calculated, and the autoencoder model to be trained is iteratively adjusted according to the loss value to obtain the decoder component.

[0017] Optionally, the autoencoder model to be trained includes a decoder component to be trained and an encoder component to be trained, the encoder component to be trained is used to encode the sample audio to obtain timbre conversion information of the sample audio, and the decoder component to be trained is used to decode and synthesize based on the timbre conversion information of the sample audio to generate the predicted audio.

[0018] Optionally, calculating the loss value between the predicted audio and the standard audio includes:

[0019] A multi-scale spectrum loss between the predicted audio and the standard audio is calculated.

[0020] Optionally, the step of extracting the timbre conversion information of the original audio includes:

[0021] Calculating loudness information of the original audio, and passing the loudness information through a first filter to obtain a loudness curve of the original audio, wherein the first filter includes a median smoothing filter;

[0022] Obtaining a time domain waveform envelope of the original audio through a second filter, wherein the second filter comprises a Hilbert filter;

[0023] The fundamental frequency of the original audio is extracted, and the loudness curve, the time domain waveform envelope, and the fundamental frequency are used as timbre conversion information of the original audio.

[0024] Optionally, the method for obtaining the sinusoidal component of the original audio further includes:

[0025] Decomposing the original audio to obtain sinusoidal audio and noise components in the original audio, wherein the sinusoidal audio includes a fundamental frequency and harmonics;

[0026] Based on the sinusoidal audio, the spectrum envelope and harmonic frequency values ​​of the original audio are obtained through third filter modeling as the sinusoidal components of the original audio, and the third filter includes a source filter model.

[0027] Optionally, the step of performing interpolation processing on the harmonic audio and the sinusoidal component by a spectrum feature interpolation method based on a preset interpolation coefficient to obtain the target audio after timbre conversion includes:

[0028] The interpolation process is performed on the harmonic audio and the sinusoidal component by a spectrum feature interpolation method based on a preset interpolation coefficient to obtain a first audio;

[0029] Adding the noise component to the first audio to obtain a target audio after timbre conversion;

[0030] The interpolation coefficient is used to control the interpolation ratio of the harmonic frequency value and the deformation degree of the spectrum envelope, so as to control the timbre deformation degree of the target audio relative to the original audio.

[0031] According to a second aspect of an embodiment of the present disclosure, there is provided a timbre conversion processing device, comprising:

[0032] A conversion information acquisition module, configured to acquire original audio and extract timbre conversion information of the original audio;

[0033] A reference audio synthesis module is configured to synthesize a reference audio with a target timbre according to the timbre conversion information, wherein the reference audio at least includes harmonic audio, and the harmonic audio is an audio that is consistent with the audio content of the original audio and only includes harmonics;

[0034] The timbre conversion processing module is configured to execute the acquisition of the sinusoidal component of the original audio, and based on the preset interpolation coefficient, perform interpolation processing on the harmonic audio and the sinusoidal component through the spectrum feature interpolation method to obtain the target audio after timbre conversion, and the timbre of the target audio is between the timbre of the original audio and the target timbre.

[0035] Optionally, the reference audio synthesis module includes:

[0036] A reference audio synthesis submodule, configured to perform synthesis of reference audio with target timbre through a decoder component according to the timbre conversion information;

[0037] The decoder component is obtained by training the autoencoder model to be trained, and the autoencoder model to be trained includes at least one decoder component to be trained.

[0038] Optionally, the device further comprises:

[0039] The decoder component training module is configured to execute the steps of acquiring at least one sample audio and a standard audio corresponding to the sample audio, wherein the standard audio has a target timbre and is consistent with the audio content of the sample audio; inputting the sample audio into the autoencoder model to be trained to obtain a predicted audio; calculating a loss value between the predicted audio and the standard audio, and iteratively adjusting the autoencoder model to be trained according to the loss value to obtain the decoder component.

[0040] Optionally, the autoencoder model to be trained includes a decoder component to be trained and an encoder component to be trained, the encoder component to be trained is used to encode the sample audio to obtain timbre conversion information of the sample audio, and the decoder component to be trained is used to decode and synthesize based on the timbre conversion information of the sample audio to generate the predicted audio.

[0041] Optionally, the decoder component training module is configured to perform:

[0042] A multi-scale spectrum loss between the predicted audio and the standard audio is calculated.

[0043] Optionally, the conversion information acquisition module includes:

[0044] A loudness curve acquisition submodule is configured to calculate the loudness information of the original audio and pass the loudness information through a first filter to obtain a loudness curve of the original audio, wherein the first filter includes a median smoothing filter;

[0045] A time domain waveform envelope acquisition submodule is configured to obtain the time domain waveform envelope of the original audio through a second filter, wherein the second filter includes a Hilbert filter;

[0046] The fundamental frequency audio acquisition submodule is configured to extract the fundamental frequency of the original audio, and use the loudness curve, the time domain waveform envelope and the fundamental frequency as the timbre conversion information of the original audio.

[0047] Optionally, the timbre conversion processing module includes:

[0048] An audio decomposition processing module is configured to perform decomposition processing on the original audio to obtain sinusoidal audio and noise components in the original audio, wherein the sinusoidal audio includes a fundamental frequency and harmonics;

[0049] The sine parameter acquisition module is configured to execute, based on the sine audio, obtaining the spectrum envelope and harmonic frequency values ​​of the original audio through a third filter modeling as the sine component of the original audio, and the third filter includes a source filter model.

[0050] Optionally, the timbre conversion processing module includes:

[0051] An interpolation processing submodule is configured to perform interpolation processing on the harmonic audio and the sinusoidal component based on a preset interpolation coefficient by using a spectrum feature interpolation method to obtain a first audio;

[0052] A noise adding submodule, configured to add the noise component to the first audio to obtain a target audio after timbre conversion;

[0053] The interpolation coefficient is used to control the interpolation ratio of the harmonic frequency value and the deformation degree of the spectrum envelope, so as to control the timbre deformation degree of the target audio relative to the original audio.

[0054] According to a third aspect of an embodiment of the present disclosure, there is provided an electronic device, including:

[0055] processor;

[0056] a memory for storing instructions executable by the processor;

[0057] The processor is configured to execute the instructions to implement any one of the timbre conversion processing methods described in the first aspect.

[0058] According to a fourth aspect of an embodiment of the present disclosure, a storage medium is provided. When instructions in the storage medium are executed by a processor of an electronic device, the electronic device can execute any one of the timbre conversion processing methods described in the first aspect.

[0059] According to a fifth aspect of an embodiment of the present disclosure, a computer program product is provided. When the instructions in the storage medium are executed by a processor of an electronic device, the electronic device can perform any one of the timbre conversion processing methods described in the first aspect.

[0060] The technical solution provided by the embodiments of the present disclosure brings at least the following beneficial effects: by synthesizing reference audio with target timbre and combining it with time-frequency domain feature interpolation technology based on auditory perception feature research, a controllable timbre conversion scheme is implemented to improve the effectiveness and flexibility of timbre conversion.

[0061] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0062] The drawings herein are incorporated into and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the description are used to explain the principles of the present disclosure, and do not constitute improper limitations on the present disclosure.

[0063] Figure 1 The figure is a flowchart of a method for tone conversion according to an exemplary embodiment.

[0064] Figure 2 is a flowchart of another timbre conversion processing method according to an exemplary embodiment.

[0065] Figure 3A It is a schematic diagram of the structure of an autoencoder model to be trained according to an exemplary embodiment.

[0066] Figure 3B The figure is a schematic diagram of a controllable timbre conversion process according to an exemplary embodiment.

[0067] Figure 4 The invention is a block diagram of a device for voice conversion according to an exemplary embodiment.

[0068] Figure 5 It is a block diagram of another timbre conversion processing device according to an exemplary embodiment.

[0069] Figure 6 The diagram is a schematic structural diagram of an electronic device according to an exemplary embodiment.

[0070] Figure 7 is a block diagram of another device according to an exemplary embodiment. DETAILED DESCRIPTION

[0071] In order to enable ordinary persons in the art to better understand the technical solutions of the present disclosure, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the accompanying drawings.

[0072] It should be noted that the terms "first", "second", etc. in the specification and claims of the present disclosure and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present disclosure described herein can be implemented in an order other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. Instead, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the appended claims.

[0073] In the first aspect, a timbre conversion processing method provided by an embodiment of the present disclosure will be introduced in detail.

[0074] Figure 1 is a flowchart of a method for tone conversion processing according to an exemplary embodiment. Figure 1 As shown, the tone conversion processing method can be used in electronic devices such as mobile phones and computers, and includes the following steps.

[0075] In step 110, original audio is acquired, and timbre conversion information of the original audio is extracted.

[0076] The original audio may be a digital audio file obtained by recording the non-polyphonic solo sound of any natural musical instrument, such as a wav (wave) file, an mp3 (Moving Picture Experts Group AudioLayer III) file, etc. Different natural musical instruments have different timbres. For example, the flutes commonly used in the current orchestra are generally made of brass or silver alloy. The timbre of such metal flutes is very different from that of the ebony flutes of the Baroque period.

[0077] The timbre conversion information is used to indicate the timbre of the original audio. For example, the timbre conversion information may include information such as fundamental frequency, loudness curve, and time domain waveform envelope.

[0078] Among them, general sounds are composed of a series of vibrations with different frequencies and amplitudes emitted by the sound-producing body. Among these vibrations, there is a vibration with the lowest frequency, and the sound emitted by it is the fundamental tone, and the rest are overtones. The sound produced by the overall vibration of the sound-producing body is called the fundamental tone, which determines the pitch. Among them, the larger the length of the fundamental tone vibration, the smaller the frequency. The fundamental tone frequency can be understood as the frequency of the fundamental tone. The loudness curve can be understood as a curve that characterizes the change in the loudness of the audio. The spectrum envelope means that the spectrum is a collection of many different frequencies, forming a very wide frequency range, and the amplitudes of different frequencies may be different. The curve formed by connecting the highest amplitude points of different frequencies is called the spectrum envelope. The time domain waveform envelope can be understood as a waveform change curve of an audio in the time domain. Harmonics are also overtones. The harmonic frequency is an integer multiple of the fundamental frequency, and the fundamental frequency is the inverse of the period of the periodic signal. The harmonic distribution of the sound signal is the main factor affecting the timbre.

[0079] In the embodiments of the present disclosure, the timbre conversion information of the original audio may be obtained in any available manner, without limitation.

[0080] In step 120, a reference audio with a target timbre is synthesized according to the timbre conversion information, wherein the reference audio at least includes harmonic audio, and the harmonic audio is audio that is consistent with the audio content of the original audio and only includes harmonics.

[0081] By processing the timbre conversion information, a reference audio with a target timbre can be obtained, wherein the target timbre is different from the timbre of the original audio. The reference audio contains harmonic audio, and the audio content of the harmonic audio is consistent with that of the original audio. For example, the original audio and the reference audio can be regarded as the sounds of two different instruments playing the same melody.

[0082] In step 130, the sinusoidal component of the original audio is obtained, and based on a preset interpolation coefficient, the harmonic audio and the sinusoidal component are interpolated by a spectral feature interpolation method to obtain a target audio after timbre conversion, and the timbre of the target audio is between the timbre of the original audio and the target timbre.

[0083] Among them, the sinusoidal components of the original audio can include the spectrum envelope and harmonic frequency values. The spectrum envelope is a curve formed by connecting the highest amplitude points of different frequencies, which can also be called the spectrum envelope line. Harmonics are also called overtones, which refer to the components greater than integer multiples of the fundamental frequency obtained by Fourier series decomposition of periodic non-sinusoidal alternating current. In audio, the role of harmonics is to beautify the sound and give it color.

[0084] In this solution, the harmonic audio and the sinusoidal component are interpolated by the spectrum feature interpolation method to obtain the target audio, and the timbre of the target audio is between the timbre of the original audio and the target timbre, thereby realizing the timbre conversion of the original audio. For example, in the post-production stage, music producers can use this solution to add effects to the timbre of musical instruments. Taking woodwind instruments as an example, the timbre conversion technology can gradually convert the timbre of the metal flute to the timbre of ebony, making the sound more rounded and bright, and more suitable for expressing pastoral melodies. Alternatively, the timbre of the clarinet can be gradually converted to the timbre of the French horn, producing a transitional timbre between the timbre of the clarinet and the timbre of the French horn, so that the natural sound of the woodwind instrument can be integrated into the metal hearing. In this way, the infinite possibilities of timbre can be explored, the boundaries of the timbre space can be expanded, and countless interesting ways of musical expression can be designed.

[0085] In the process of interpolating the harmonic audio and the sine component, the original audio can be gradually converted from its original timbre to another timbre to obtain an audio with any transitional timbre between the original timbre and another timbre, wherein the other timbre is the target timbre mentioned above.

[0086] Specifically, the timbre conversion can be performed on the sinusoidal components of the original audio and the harmonic audio of the reference audio based on the characteristic interpolation algorithm. The interpolation factor used in the spectrum characteristic interpolation method can be defined as α, which is used to control the interpolation ratio of the harmonic frequency value and the deformation degree of the spectrum envelope, thereby controlling the timbre deformation process, that is, by controlling the interpolation coefficient, the timbre conversion degree of the original audio can be controlled, so that any timbre between the original timbre and the target timbre can be obtained.

[0087] Among them, the spectral feature interpolation method (spectral morphing) mainly includes two parts: harmonic frequency value interpolation and spectral envelope interpolation.

[0088] Harmonic frequency value interpolation refers to the interpolation algorithm between the harmonic frequency value of the original audio and the harmonic frequency value of the synthesized reference audio. The interpolation algorithm only acts on the frequency coordinates of the spectrum to form the distribution of the harmonic sequence of the converted sound in frequency. For example, the harmonic frequency value interpolation method can be implemented using the following formula:

[0089]

[0090] Among them, α is the interpolation coefficient, which is used to set the interpolation ratio of the two sound parameters, thereby controlling the timbre of the converted sound. n is the sequence number of the harmonic, f nα is the harmonic value of the interpolated sound, f n1 and f n2The first sound and the second sound can be the original audio and the reference audio respectively. Of course, the first sound can also be set as the reference audio and the second sound as the original audio according to the needs, which is not limited in the embodiment of the present disclosure.

[0091] The spectrum envelope interpolation can be expressed using line spectral frequencies (LSF). For each frame, the deformed frequency envelope gives the amplitude at each interpolated harmonic frequency value position to form a deformed amplitude spectrum. In addition, the amplitude spectrum can be inversely transformed according to the time domain waveform envelope extracted from the original audio. If the waveform envelope difference between the original audio and the reference audio is relatively large, the waveform envelope of the converted audio can be adjusted according to the preset ADSR envelope, which is not limited in the embodiments of the present disclosure.

[0092] Reference Figure 2 In the embodiment of the present disclosure, step 120 may further include:

[0093] Step 121, synthesizing reference audio with target timbre through a decoder component according to the timbre conversion information; wherein the decoder component is obtained by training the autoencoder model to be trained, and the autoencoder model to be trained includes at least one decoder component to be trained.

[0094] In the disclosed embodiment, in order to facilitate the synthesis of accurate reference audio, an autoencoder model can be pre-trained, and the autoencoder model is used to realize the mapping relationship between timbre conversion information and harmonic amplitude distribution and noise transfer function. The trained autoencoder model can convert the input original audio into a reference audio with the timbre personality of the reference audio for output.

[0095] In this way, the decoder component can be used to achieve non-parallel conversion between original audio and reference audio, where "non-parallel" means that the original sound and target sound matching data do not need to have the same audio content. That is, there is no need to pre-acquire audio data with the target timbre and the same audio content as the original audio. The timbre of the original audio can be directly converted and the audio content of the original audio can be retained to obtain reference audio with the target timbre, thereby improving the generalization ability of the conversion processing method technology.

[0096] Among them, the structure of the autoencoder model to be trained can be customized according to needs, which is not limited in the embodiments of the present disclosure. However, in order to facilitate the subsequent use of the trained autoencoder model to be trained to synthesize audio with a target timbre, it is possible to set the autoencoder model to be trained to include at least one decoder component to be trained. After the autoencoder model to be trained is trained, the trained decoder component can be used to synthesize a reference audio with a target timbre. The training sample of the autoencoder model to be trained can be a section of audio with a target timbre. The decoder component can be any machine learning model with a decoding function, for example, it can be any neural network structure, which is not limited in the embodiments of the present disclosure.

[0097] Reference Figure 2 In the embodiment of the present disclosure, the timbre conversion processing method may further include:

[0098] Step 210, obtaining at least one sample audio and a standard audio corresponding to the sample audio, wherein the standard audio has a target timbre and is consistent with the audio content of the sample audio; inputting the sample audio into the autoencoder model to be trained to obtain a predicted audio; calculating a loss value between the predicted audio and the standard audio, iteratively adjusting the autoencoder model to be trained according to the loss value, and obtaining a decoder component.

[0099] In this way, through iterative adjustment of the trained autoencoder model, the difference between the predicted audio output by the decoder component based on the sample audio and the standard audio corresponding to the sample audio is already very small. Then, in the application process of the decoder component, the reference audio output by the decoder component based on the original audio can also be considered to have the target timbre and be consistent with the audio content of the original audio, and non-parallel conversion between the original audio and the reference audio can be achieved.

[0100] Among them, the autoencoder model to be trained includes a decoder component to be trained and an encoder component to be trained. The encoder component to be trained is used to encode the sample audio to obtain the timbre conversion information of the sample audio; the decoder component to be trained is used to decode and synthesize based on the timbre conversion information of the sample audio to generate a standard audio corresponding to the sample audio.

[0101] In this way, in the process of training the autoencoder model, the decoder component to be trained is also trained accordingly to obtain a decoder component. In practical applications, the decoder component can process the timbre conversion information of the original audio to obtain a reference audio corresponding to the original audio. The reference audio has the target timbre and is consistent with the audio content of the original audio.

[0102] like Figure 3A FIG. 1 is a schematic diagram of an autoencoder model to be trained in an embodiment of the present disclosure. Figure 3AThe combination of a decoder to be trained (decoder) and an audio synthesizer (Synthesizer) shown in the figure can be understood as a decoder to be trained component, and the encoder to be trained (training encoder) can be understood as an encoder to be trained component.

[0103] The training sample of the autoencoder model to be trained may include at least one sample audio with a target timbre. For example, a set of solo music with a total length of about one hour and a target timbre may be selected as the training set. The solo music may be recorded in a digital audio file format in the same recording environment.

[0104] In the training process of the autoencoder model to be trained, the sample audio is used as input and sequentially passes through the encoder component to be trained and the decoder component to be trained to obtain the predicted audio. By comparing the standard audio and the predicted audio corresponding to the sample audio, the loss of the autoencoder model to be trained can be obtained, thereby adjusting the parameters in the autoencoder model to be trained. The loss function form of the autoencoder model to be trained can be customized according to the needs, which is not limited in the embodiments of the present disclosure.

[0105] Optionally, in the embodiment of the present disclosure, calculating the loss value between the predicted audio and the standard audio in step 210 may specifically be calculating a multi-scale spectrum loss between the predicted audio and the standard audio.

[0106] Among them, the multi-scale spectral loss between the predicted audio and the standard audio is obtained by minimizing the amplitude spectrum of the predicted audio and the standard audio at different time domain resolutions. The scale of the time domain resolution can be customized according to needs, which is not limited in the embodiment of the present disclosure. Due to the adoption of a multi-scale loss function, the decoder component finally trained has a stronger generalization ability, and the timbre of the reference audio output by the decoder component is closer to the target timbre.

[0107] For example, the multi-scale spectrum loss can be calculated at three short-time Fourier scales, namely 128 Fourier points, 512 Fourier points and 2048 Fourier points. The frame lengths corresponding to the three Fourier scales are 5ms, 20ms and 120ms respectively, and the overlap rate can be 75%.

[0108] The multi-scale spectrum loss can be calculated using the following formula:

[0109]

[0110] Loss = L 128 +L 512 +L 2048

[0111] Among them, L i represents the spectrum loss at the Fourier point scale, S i (n) represents the spectrum of the standard audio at the i-th Fourier point scale, represents the spectrum of the predicted audio at the i Fourier point scale, n represents the discrete time in the frequency domain, Loss represents the multi-scale spectrum loss, L 128 , L 512 , L 2048 They represent the spectrum loss between the predicted audio and the standard audio at the scales of 128 Fourier points, 512 Fourier points, and 2048 Fourier points, respectively.

[0112] also, Figure 3A The encoder to be trained can output the time-frequency domain parameters (temporal spectral feature) of the original audio. The time-frequency domain parameters may include fundamental frequency, loudness curve and MFCC (Mel Frequency Cepstrum Coefficient), etc. Among them, MFCC can be used for dimensionality reduction and hidden layer feature extraction through a 12-layer residual network.

[0113] Figure 3A The decoder to be trained can be a digital harmonic oscillator. The digital harmonic oscillator generates a linear combination x(n) of K sine wave sequences at each discrete time n. The sine wave includes the fundamental frequency and the harmonic wave. The calculation process is shown in the following formula:

[0114]

[0115] A k (n) = A(n)C k (n)

[0116] Among them, the instantaneous frequency of the sine wave of the sample audio is φ k (n) can be an integer multiple of the fundamental frequency of the sine wave, and the goal of training the autoencoder is to form the target timbre and the time-varying amplitude A of the sine wave sequence k (n), the preset coefficient A(n) is the overall loudness of the sound, which can be extracted from the original audio by the encoder in the decoder component to be trained, or can be set by relevant personnel. k (n) is the amplitude change of the sine wave of the sample audio after normalization distribution, C k (n) satisfies the non-negative property, and the normalized amplitude C of k sine waves k The cumulative sum of (n) is 1. After training, C k The (n) set represents the energy distribution of the harmonics. The C of k sine wavesk (n) and the corresponding frequency value φ k (n) jointly determine the timbre personality of the output audio. The specific value of K can be customized according to the needs, and this embodiment of the disclosure is not limited to this.

[0117] For example, assuming that the autoencoder model to be trained is a DDSP (differentiable digital signal processing) autoencoder, the DDSP autoencoder can choose to predict the filtered noise of natural acoustic instruments. To obtain interpretability and prevent phase distortion, the decoder predicts the frequency domain transfer function of the time-varying FIR (Finite Impulse Response) filter for each output frame. Finally, the filtered noise response is assigned to the harmonic audio to obtain the predicted audio.

[0118] Reference Figure 2 In the embodiment of the present disclosure, step 110 may further include:

[0119] Step 111, calculating loudness information of the original audio, and passing the loudness information through a first filter to obtain a loudness curve of the original audio, wherein the first filter includes a median smoothing filter;

[0120] Step 112, obtaining a time domain waveform envelope of the original audio through a second filter, where the second filter includes a Hilbert filter;

[0121] Step 113, extracting the fundamental frequency of the original audio, and using the loudness curve, the time domain waveform envelope, and the fundamental frequency as the timbre conversion information of the original audio.

[0122] In this way, by extracting the timbre conversion information such as the loudness curve, time domain waveform envelope and fundamental frequency of the original audio, the synthesized reference audio can be made consistent with the audio content of the original audio. Therefore, in the timbre conversion process, there is no need to pre-acquire audio with a target timbre that is consistent with the audio content of the original audio, thereby improving the generalization and flexibility of the timbre conversion.

[0123] Among them, loudness information is different from the time domain waveform envelope. Loudness information belongs to the frequency domain perception feature. Different weights will be given to the amplitude in different frequency bands. The extraction of loudness information serves the decoder component. In the embodiment of the present disclosure, the time domain waveform envelope analysis result needs to restore the signal curve of the original audio as much as possible. The purpose of time domain waveform envelope extraction is to serve the timbre deformation.

[0124] When calculating the loudness information of the original audio, the loudness of the original audio can be weighted. Weighting means that a certain value has been modified according to certain rules. Since the human ear is particularly sensitive to the intermediate frequency, if the signal-to-noise ratio of an amplifier in the intermediate frequency band is large enough, then even if the signal-to-noise ratio is slightly lower in the low and high frequency bands, the human ear is not easy to detect. There are generally three weighting methods: A, B, and C. The A-weighted sound level simulates the loudness of the human ear to 40-square pure tone. When the signal passes through, its low frequency and mid-band frequency (below 1000Hz) have a large attenuation; the B-weighted sound level simulates the loudness of the human ear to 70-square pure tone, and it has a certain attenuation on the low frequency band of the signal; the C-weighted sound level simulates the loudness of the human ear to 100-square pure tone, and has a nearly flat response in the entire frequency range. The main difference between the three is the degree of attenuation of the low-frequency components of the noise. A has the most attenuation, B is the second, and C is the least.

[0125] In the embodiment of the present disclosure, in order to make the processed audio loudness closer to the human ear's perception of loudness, the audio loudness can preferably be processed in an A-weighting manner. In addition, after the A-weighting process, the audio can also pass through a first filter to reduce the impact of the release time period, and the first filter can include but is not limited to a median smoothing filter.

[0126] In addition, in the disclosed embodiment, the time domain waveform envelope of the audio can also be obtained by a second filter, and the second filter can include but is not limited to a Hilbert filter. The fundamental frequency of the original audio can be extracted by a probabilistic YIN algorithm.

[0127] Reference Figure 2 In the embodiment of the present disclosure, step 130 of the timbre conversion method obtains the sinusoidal component of the original audio, and may also include:

[0128] Step 310, decomposing the original audio to obtain sinusoidal audio and noise components in the original audio, where the sinusoidal audio includes fundamental frequency and harmonics;

[0129] Step 320: Based on the sinusoidal audio, the spectrum envelope and harmonic frequency values ​​of the original audio are obtained by modeling with a third filter as the sinusoidal components of the original audio. The third filter includes a source filter model.

[0130] Among them, the sinusoidal component of the original audio may include the spectrum envelope and harmonic frequency value of the original audio, which can reflect the timbre of the original audio. In this way, by obtaining the sinusoidal component of the original audio, the original audio and the reference audio can be further interpolated to achieve flexible adjustment of the timbre of the target audio between the original audio and the reference audio, thereby improving the generalization of the timbre conversion method.

[0131] It is understandable that the noise component contained in the sound of natural musical instruments also plays a key role in the auditory recognition of timbre. Therefore, in one implementation, the input original audio can also be decomposed into two parts: a sine component and a noise component, and then the spectrum envelope of the noise component is extracted, and the sine component of the original audio, the spectrum envelope of the original audio noise component and the reference audio are combined to perform audio conversion to obtain the target audio.

[0132] In the embodiment of the present disclosure, the third filter may include but is not limited to a source filter model (source-filtermodel), wherein, in the face of sinusoidal audio, the source filter model may use the TE (Temporal Evenlope, time domain envelope) method to establish a time-varying spectrum envelope, and use the peak-picking algorithm to select the time-varying frequency value of the harmonic. In the face of the remaining noise components, the spectrum envelope of the noise may be estimated by a linear prediction method, which is not limited in the embodiment of the present disclosure.

[0133] For the reference audio output by the decoder component, usually only the harmonic audio therein is used for timbre conversion. It is possible to choose to analyze the spectrum envelope and harmonic frequency value of the harmonic audio at the output end of the decoder component, or it is possible to choose to analyze the spectrum envelope and harmonic frequency value of the harmonic audio as the second group of input sound callback sine and noise decomposition components after completing the timbre conversion task. The specific selection method can depend on the synthesis quality of different audios by the decoder component, which is not limited in the embodiments of the present disclosure.

[0134] In addition, when the spectrum envelope is obtained through the source filter model, the data structure of the harmonic audio from the decoder component can be reconstructed before interpolation processing to match the function definition of the source filter model, which is not limited by the embodiments of the present disclosure.

[0135] Reference Figure 2 In the embodiment of the present disclosure, in step 130, based on the preset interpolation coefficient, the harmonic audio and the sinusoidal component are interpolated by a spectrum feature interpolation method to obtain the target audio after timbre conversion, which may further include:

[0136] Step 131, based on a preset interpolation coefficient, interpolate the harmonic audio and the sinusoidal component by a spectrum feature interpolation method to obtain a first audio;

[0137] Step 132, adding a noise component to the first audio to obtain a target audio after timbre conversion;

[0138] The interpolation coefficient is used to control the interpolation ratio of the harmonic frequency value and the deformation degree of the spectrum envelope, so as to control the timbre deformation degree of the target audio relative to the original audio.

[0139] As mentioned above, the noise component contained in the sound of natural musical instruments also plays a key role in the auditory recognition of timbre. Therefore, in the embodiment of the present disclosure, after the synthesized harmonic audio and the sinusoidal audio in the original audio are interpolated, in order to ensure that the audio after timbre conversion has the same auditory recognition in terms of noise component, the first audio obtained by interpolation processing can be further assigned the noise component in the original audio to obtain the target audio after timbre conversion, so that the timbre of the target audio is more realistic and the timbre conversion effect is better.

[0140] like Figure 3B The figure shows a schematic diagram of a controllable timbre conversion process. The input audio can be divided into two paths: one path is synthesized into a reference audio with a target timbre through a decoder component, and then its frequency domain features are extracted from the reference audio; the other path is decomposed and extracted into time and frequency domain features through digital signal processing. Finally, based on the two frequency domain features, a feature interpolation algorithm is performed to generate the target audio after timbre conversion.

[0141] The detailed process of each processing route is as follows:

[0142] First path: The input raw audio is sent to the Loudness & Envelope Extraction and Pitch Tracker components of the encoder to extract timbre conversion information. The time domain processing component is used to extract the tenporal envelope and loudness curve, and the pitch tracking component is used to extract the fundamental frequency.

[0143] Afterwards, the decoder and synthesizer components (Decoder & Synthesizer) synthesize reference audio with the target timbre based on the timbre conversion information, and the reference audio includes harmonic audio. The harmonic audio is consistent with the input original audio in terms of musical content and imitates the target timbre as much as possible in terms of timbre auditory perception.

[0144] In the disclosed embodiment, all data processing tasks involving the deep learning model can be packaged in the decoder component structure and deployed on the server, and other sound signal processing tasks are placed on the client. For the harmonic audio from the decoder component, before interpolation, its data structure can be reconstructed (Note Segmentation and Reshape) to match the function definition of the source filter model.

[0145] The second path is to decompose the input original audio into two parts, a sinusoidal audio and a noise component, through a sinusoidal and residual decomposition component. After that, the sinusoidal audio is processed by a source filter model to obtain the sinusoidal component, that is, the spectral envelope (Sinusoidal Spectral envelope) and the harmonic frequency value (partial frequency values) of the sinusoidal audio. Optionally, in order to further improve the effect of the timbre conversion, the noise component can also be processed by a source filter model to obtain the spectral envelope (Noise Spectral envelope) of the noise component. Among them, the noise spectral envelope is an optional part and may not be obtained, which is not limited to the embodiment of the present disclosure.

[0146] Furthermore, the spectral feature interpolation method can be used to interpolate the harmonic audio in the reference audio synthesized by the decoder component and the sinusoidal components of the original audio (spectral morphing), and the spectral envelope of the original audio noise component can be further combined to obtain the target audio after timbre deformation.

[0147] In addition, if single notes are processed, there is no need for segmentation. If music is processed, before modeling and interpolation methods, the notes can be segmented according to the changes in pitch as required, and the corresponding spectrum envelope is calculated for each segment after segmentation.

[0148] From the above, it can be seen that in the technical solution provided by the embodiment of the present disclosure, a controllable timbre conversion scheme is realized by synthesizing reference audio with target timbre and combining it with time-frequency domain feature interpolation technology based on auditory perception feature research, thereby improving the effectiveness and flexibility of timbre conversion.

[0149] Specifically, in the disclosed embodiment, a timbre conversion architecture that integrates an autoencoder neural network model and a feature interpolation algorithm and has controllable capabilities is proposed. Moreover, in the training problem of the autoencoder model, the idea of ​​reconstructing the training set is adopted to make the predicted audio output by the autoencoder model as similar as possible to the standard audio corresponding to the training set data in the Fourier spectrum, thereby meeting the non-parallel requirements and making up for the problem that the sound deformation technology of the signal processing system can only process given parallel notes. In the execution problem of timbre deformation, the disclosed embodiment always performs spectrum modeling, prediction and synthesis around the harmonic components of the original audio. The sound of musical instruments with fundamental frequency and harmonic wave rules will be decomposed into parameter matrices such as overtone distribution sequence and time-spectrum envelope by the spectrum modeling framework, and the decoupled parameters have interpretability and acoustic commonality. Therefore, the execution program has a many-to-one sound processing capability. Therefore, the solution provided by the disclosed embodiment is controllable and interpretable, and can meet the high requirements of music information processing tasks for changes in complex perceptual features such as pitch, sound intensity, and sound length.

[0150] In the second aspect, a timbre conversion processing device provided by an embodiment of the present disclosure will be introduced in detail.

[0151] Figure 4 is a block diagram of a timbre conversion processing device according to an exemplary embodiment. Figure 4 The device includes a conversion information acquisition module 410, a reference audio synthesis module 420 and a timbre conversion processing module 430.

[0152] The conversion information acquisition module 410 is configured to acquire the original audio and extract the timbre conversion information of the original audio;

[0153] The reference audio synthesis module 420 is configured to synthesize a reference audio with a target timbre according to the timbre conversion information, wherein the reference audio at least includes harmonic audio, and the harmonic audio is an audio that is consistent with the audio content of the original audio and only includes harmonics;

[0154] The timbre conversion processing module 430 is configured to execute the acquisition of the sinusoidal component of the original audio, and based on the preset interpolation coefficient, perform interpolation processing on the harmonic audio and the sinusoidal component through the spectrum feature interpolation method to obtain the target audio after timbre conversion, and the timbre of the target audio is between the timbre of the original audio and the target timbre.

[0155] Reference Figure 5 In the embodiment of the present disclosure, the reference audio synthesis module 420 includes:

[0156] The reference audio synthesis submodule 421 is configured to perform synthesis of reference audio with target timbre through a decoder component according to the timbre conversion information;

[0157] The decoder component is obtained by training the autoencoder model to be trained, and the autoencoder model to be trained includes at least one decoder component to be trained.

[0158] Reference Figure 5 In the embodiment of the present disclosure, the device may further include:

[0159] The decoder component training module 510 is configured to execute the steps of obtaining at least one sample audio and a standard audio corresponding to the sample audio, wherein the standard audio has a target timbre and is consistent with the audio content of the sample audio; inputting the sample audio into the autoencoder model to be trained to obtain a predicted audio; calculating a loss value between the predicted audio and the standard audio, and iteratively adjusting the autoencoder model to be trained according to the loss value to obtain the decoder component.

[0160] Optionally, the autoencoder model to be trained includes a decoder component to be trained and an encoder component to be trained, the encoder component to be trained is used to encode the sample audio to obtain timbre conversion information of the sample audio, and the decoder component to be trained is used to decode and synthesize based on the timbre conversion information of the sample audio to generate the predicted audio.

[0161] Optionally, in the embodiment of the present disclosure, the decoder component training module 510 is configured to execute:

[0162] A multi-scale spectrum loss between the predicted audio and the standard audio is calculated.

[0163] Reference Figure 5 In the embodiment of the present disclosure, the conversion information acquisition module 410 may include:

[0164] A loudness curve acquisition submodule 411 is configured to calculate the loudness information of the original audio and pass the loudness information through a first filter to obtain a loudness curve of the original audio, wherein the first filter includes a median smoothing filter;

[0165] The time domain waveform envelope acquisition submodule 412 is configured to obtain the time domain waveform envelope of the original audio through a second filter, wherein the second filter includes a Hilbert filter;

[0166] The fundamental frequency audio acquisition submodule 413 is configured to extract the fundamental frequency of the original audio by executing a probabilistic YIN algorithm, and use the loudness curve, the time domain waveform envelope and the fundamental frequency as the timbre conversion information of the original audio.

[0167] Reference Figure 5In the disclosed embodiment, the timbre conversion processing module includes:

[0168] The audio decomposition processing module 610 is configured to perform decomposition processing on the original audio to obtain sinusoidal audio and noise components in the original audio, wherein the sinusoidal audio includes a fundamental frequency and harmonics;

[0169] The sinusoidal parameter acquisition module 620 is configured to execute, based on the sinusoidal audio, obtaining the spectrum envelope and harmonic frequency values ​​of the original audio as the sinusoidal components of the original audio through a third filter modeling, wherein the third filter includes a source filter model.

[0170] Reference Figure 5 In the embodiment of the present disclosure, the timbre conversion processing module 430 may include:

[0171] The interpolation processing submodule 431 is configured to perform interpolation processing on the harmonic audio and the sinusoidal component based on a preset interpolation coefficient by using a spectrum feature interpolation method to obtain a first audio;

[0172] The noise adding submodule 432 is configured to add the noise component to the first audio to obtain a target audio after timbre conversion;

[0173] The interpolation coefficient is used to control the interpolation ratio of the harmonic frequency value and the deformation degree of the spectrum envelope, so as to control the timbre deformation degree of the target audio relative to the original audio.

[0174] From the above, it can be seen that in the technical solution provided by the embodiment of the present disclosure, a controllable timbre conversion scheme is realized by synthesizing reference audio with target timbre and combining it with time-frequency domain feature interpolation technology based on auditory perception feature research, thereby improving the effectiveness and flexibility of timbre conversion.

[0175] Regarding the device in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0176] In a third aspect, an electronic device provided by an embodiment of the present disclosure will be introduced in detail.

[0177] like Figure 6 As shown, a schematic diagram of the structure of an electronic device provided by an embodiment of the present disclosure.

[0178] In an exemplary embodiment, a computer-readable storage medium including instructions is also provided, such as a memory including instructions, and the above instructions can be executed by a processor of an electronic device to perform the above method. Optionally, the computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0179] In an exemplary embodiment, a computer program product is also provided, which, when executed on a computer, enables the computer to implement the above-mentioned information pushing method.

[0180] Figure 7 7 is a block diagram of a device 700 for voice conversion processing according to an exemplary embodiment. For example, the device 700 may be a mobile phone, a computer, a digital broadcast terminal, a messaging device, a game console, a tablet device, a medical device, a fitness device, a personal digital assistant, etc.

[0181] Reference Figure 7 , the device 700 may include one or more of the following components: a processing component 702 , a memory 704 , a power component 706 , a multimedia component 708 , an audio component 710 , an input / output (I / O) interface 712 , a sensor component 714 , and a communication component 716 .

[0182] The processing component 702 generally controls the overall operation of the device 700, such as operations associated with display, phone calls, data communications, camera operations, and recording operations. The processing component 702 may include one or more processors 720 to execute instructions to complete all or part of the steps of the above-mentioned method. In addition, the processing component 702 may include one or more modules to facilitate the interaction between the processing component 702 and other components. For example, the processing component 702 may include a multimedia module to facilitate the interaction between the multimedia component 708 and the processing component 702.

[0183] The memory 704 is configured to store various types of data to support operations on the device 700. Examples of such data include instructions for any application or method operating on the device 700, contact data, phone book data, messages, pictures, videos, etc. The memory 704 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0184] The power supply component 706 provides power to the various components of the device 700. The power supply component 706 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power for the device 700.

[0185] The multimedia component 708 includes a screen that provides an output interface between the device 700 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The touch sensor may not only sense the boundaries of the touch or slide action, but also detect the duration and pressure associated with the touch or slide operation. In some embodiments, the multimedia component 708 includes a front camera and / or a rear camera. When the device 700 is in an operating mode, such as a shooting mode or a video mode, the front camera and / or the rear camera may receive external multimedia data. Each front camera and rear camera may be a fixed optical lens system or have a focal length and optical zoom capability.

[0186] The audio component 710 is configured to output and / or input audio signals. For example, the audio component 710 includes a microphone (MIC), and when the device 700 is in an operating mode, such as a call mode, a recording mode, and a speech recognition mode, the microphone is configured to receive an external audio signal. The received audio signal can be further stored in the memory 704 or sent via the communication component 716. In some embodiments, the audio component 710 also includes a speaker for outputting audio signals.

[0187] I / O interface 712 provides an interface between processing component 702 and peripheral interface modules, such as keyboards, click wheels, buttons, etc. These buttons may include but are not limited to: a home button, a volume button, a start button, and a lock button.

[0188] The sensor assembly 714 includes one or more sensors for providing various aspects of status assessment for the device 700. For example, the sensor assembly 714 can detect the open / closed state of the device 700, the relative positioning of components, such as the display and keypad of the device 700, and the sensor assembly 714 can also detect the position change of the device 700 or a component of the device 700, the presence or absence of user contact with the device 700, the orientation or acceleration / deceleration of the device 700, and the temperature change of the device 700. The sensor assembly 714 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact. The sensor assembly 714 may also include an optical sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, the sensor assembly 714 may also include an accelerometer, a gyroscope sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.

[0189] The communication component 716 is configured to facilitate wired or wireless communication between the device 700 and other devices. The device 700 can access a wireless network based on a communication standard, such as WiFi, a carrier network (such as 2G, 7G, 4G or 5G), or a combination thereof. In an exemplary embodiment, the communication component 716 receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 716 also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology and other technologies.

[0190] In an exemplary embodiment, the apparatus 700 may be implemented by one or more application specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors or other electronic components to perform the above methods.

[0191] In an exemplary embodiment, a storage medium including instructions is also provided, such as a memory 704 including instructions, and the instructions can be executed by a processor 720 of the device 700 to complete the above method. Optionally, the storage medium can be a non-transitory computer-readable storage medium, for example, the non-transitory computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0192] According to a fourth aspect of an embodiment of the present disclosure, a storage medium is provided. When instructions in the storage medium are executed by a processor of an electronic device, the electronic device is enabled to perform any of the above-mentioned timbre conversion processing methods.

[0193] According to a fifth aspect of the embodiments of the present disclosure, there is provided a computer program product comprising instructions, which, when executed on a computer, enables the computer to implement any of the above-mentioned timbre conversion processing methods.

[0194] From the above, it can be seen that in the technical solution provided by the embodiment of the present disclosure, a controllable timbre conversion scheme is realized by synthesizing reference audio with target timbre and combining it with time-frequency domain feature interpolation technology based on auditory perception feature research, thereby improving the effectiveness and flexibility of timbre conversion.

[0195] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The specification and examples are intended to be exemplary only, and the true scope and spirit of the present disclosure are indicated by the following claims.

[0196] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. A timbre conversion processing method, characterized in that: include: Acquire original audio, and extract timbre conversion information of the original audio; The step of extracting the timbre conversion information of the original audio comprises: Calculating loudness information of the original audio, and passing the loudness information through a first filter to obtain a loudness curve of the original audio, wherein the first filter includes a median smoothing filter; Obtaining a time domain waveform envelope of the original audio through a second filter, wherein the second filter comprises a Hilbert filter; Extracting the fundamental frequency of the original audio, and using the loudness curve, the time domain waveform envelope, and the fundamental frequency as timbre conversion information of the original audio; Synthesize a reference audio with a target timbre according to the timbre conversion information, wherein the reference audio at least includes harmonic audio, and the harmonic audio is an audio that is consistent with the audio content of the original audio and only includes harmonics; The sinusoidal component of the original audio is obtained, and based on a preset interpolation coefficient, the harmonic audio and the sinusoidal component are interpolated by a spectral feature interpolation method to obtain a target audio after timbre conversion, wherein the timbre of the target audio is between the timbre of the original audio and the target timbre.

2. The method according to claim 1, characterized in that: The step of synthesizing reference audio with target timbre according to the timbre conversion information comprises: synthesizing reference audio with a target timbre through a decoder component according to the timbre conversion information; The decoder component is obtained by training the autoencoder model to be trained, and the autoencoder model to be trained includes at least one decoder component to be trained.

3. The method according to claim 2, characterized in that Before the step of synthesizing reference audio with target timbre through a decoder component according to the timbre conversion information, the method further includes: Acquire at least one sample audio and a standard audio corresponding to the sample audio, wherein the standard audio has a target timbre and is consistent with the audio content of the sample audio; Inputting the sample audio into the autoencoder model to be trained to obtain predicted audio; A loss value between the predicted audio and the standard audio is calculated, and the autoencoder model to be trained is iteratively adjusted according to the loss value to obtain the decoder component.

4. The method according to claim 3, characterized in that The autoencoder model to be trained includes a decoder component to be trained and an encoder component to be trained. The encoder component to be trained is used to encode the sample audio to obtain the timbre conversion information of the sample audio, and the decoder component to be trained is used to decode and synthesize based on the timbre conversion information of the sample audio to generate the predicted audio.

5. The method according to claim 3, characterized in that: The calculating the loss value between the predicted audio and the standard audio includes: A multi-scale spectrum loss between the predicted audio and the standard audio is calculated.

6. The method according to any one of claims 1 to 5, characterized in that The obtaining of the sinusoidal component of the original audio comprises: Decomposing the original audio to obtain sinusoidal audio and noise components in the original audio, wherein the sinusoidal audio includes a fundamental frequency and harmonics; Based on the sinusoidal audio, the spectrum envelope and harmonic frequency values ​​of the original audio are obtained through third filter modeling as the sinusoidal components of the original audio, and the third filter includes a source filter model.

7. The method according to claim 6, characterized in that The step of interpolating the harmonic audio and the sinusoidal component by a spectrum feature interpolation method based on a preset interpolation coefficient to obtain the target audio after timbre conversion includes: Based on a preset interpolation coefficient, the harmonic audio and the sinusoidal component are interpolated by a spectrum feature interpolation method to obtain a first audio; Adding the noise component to the first audio to obtain a target audio after timbre conversion; The interpolation coefficient is used to control the interpolation ratio of the harmonic frequency value and the deformation degree of the spectrum envelope, so as to control the timbre deformation degree of the target audio relative to the original audio.

8. A timbre conversion processing device, characterized in that: include: A conversion information acquisition module, configured to acquire original audio and extract timbre conversion information of the original audio; The conversion information acquisition module includes: A loudness curve acquisition submodule is configured to calculate the loudness information of the original audio and pass the loudness information through a first filter to obtain a loudness curve of the original audio, wherein the first filter includes a median smoothing filter; A time domain waveform envelope acquisition submodule is configured to obtain the time domain waveform envelope of the original audio through a second filter, wherein the second filter includes a Hilbert filter; A fundamental frequency audio acquisition submodule is configured to extract the fundamental frequency of the original audio, and use the loudness curve, the time domain waveform envelope and the fundamental frequency as the timbre conversion information of the original audio; A reference audio synthesis module is configured to synthesize a reference audio with a target timbre according to the timbre conversion information, wherein the reference audio at least includes harmonic audio, and the harmonic audio is an audio that is consistent with the audio content of the original audio and only includes harmonics; The timbre conversion processing module is configured to execute the acquisition of the sinusoidal component of the original audio, and based on the preset interpolation coefficient, perform interpolation processing on the harmonic audio and the sinusoidal component through the spectrum feature interpolation method to obtain the target audio after timbre conversion, and the timbre of the target audio is between the timbre of the original audio and the target timbre.

9. The device according to claim 8, characterized in that The reference audio synthesis module comprises: A reference audio synthesis submodule, configured to perform synthesis of reference audio with target timbre through a decoder component according to the timbre conversion information; The decoder component is obtained by training the autoencoder model to be trained, and the autoencoder model to be trained includes at least one decoder component to be trained.

10. The device according to claim 9, characterized in that The device also includes: The decoder component training module is configured to execute the steps of acquiring at least one sample audio and a standard audio corresponding to the sample audio, wherein the standard audio has a target timbre and is consistent with the audio content of the sample audio; inputting the sample audio into the autoencoder model to be trained to obtain a predicted audio; calculating a loss value between the predicted audio and the standard audio, and iteratively adjusting the autoencoder model to be trained according to the loss value to obtain the decoder component.

11. The device according to claim 10, characterized in that The autoencoder model to be trained includes a decoder component to be trained and an encoder component to be trained. The encoder component to be trained is used to encode the sample audio to obtain the timbre conversion information of the sample audio, and the decoder component to be trained is used to decode and synthesize based on the timbre conversion information of the sample audio to generate the predicted audio.

12. The device according to claim 10, characterized in that The decoder component training module is configured to perform: A multi-scale spectrum loss between the predicted audio and the standard audio is calculated.

13. The device according to any one of claims 8 to 12, characterized in that The timbre conversion processing module comprises: An audio decomposition processing module is configured to perform decomposition processing on the original audio to obtain sinusoidal audio and noise components in the original audio, wherein the sinusoidal audio includes a fundamental frequency and harmonics; The sine parameter acquisition module is configured to execute, based on the sine audio, obtaining the spectrum envelope and harmonic frequency values ​​of the original audio through a third filter modeling as the sine component of the original audio, and the third filter includes a source filter model.

14. The device according to claim 13, characterized in that The timbre conversion processing module comprises: An interpolation processing submodule is configured to perform interpolation processing on the harmonic audio and the sinusoidal component based on a preset interpolation coefficient by using a spectrum feature interpolation method to obtain a first audio; A noise adding submodule, configured to add the noise component to the first audio to obtain a target audio after timbre conversion; The interpolation coefficient is used to control the interpolation ratio of the harmonic frequency value and the deformation degree of the spectrum envelope, so as to control the timbre deformation degree of the target audio relative to the original audio.

15. An electronic device, characterized in that: include: processor; a memory for storing instructions executable by the processor; The processor is configured to execute the instructions to implement the timbre conversion processing method as described in any one of claims 1 to 7.

16. A computer-readable storage medium, characterized in that: When the instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is enabled to perform the timbre conversion method according to any one of claims 1 to 7.

17. A computer program product comprising computer instructions, characterized in that When the computer instructions are executed by a processor, the timbre conversion method according to any one of claims 1 to 7 is implemented.

Images