A musical instrument timbre conversion method based on time-frequency feature refinement

Abstract

The application discloses a musical instrument tone color conversion method based on time-frequency feature refinement, which comprises the following steps: tone color feature extraction is performed on a musical instrument audio data set to establish a target tone color feature library; according to the time domain feature of the target musical instrument tone color, base frequency adjustment, tone length modification, time domain amplitude modulation and pitch synchronous interpolation are performed on the single tone audio to be converted to realize time domain feature conversion; according to the frequency domain feature of the target musical instrument tone color, the harmonic energy ratio and the fractional harmonic energy ratio of the single tone audio after the time domain feature conversion are adjusted to realize frequency domain feature conversion; the single tone audio with the target musical instrument tone color can be converted through the time-frequency feature conversion; and tone color conversion is performed on all single tone audios of the target musical instrument. The application realizes a target musical instrument tone color conversion method capable of extracting transition tone color.

Description

Technical Field

[0001] This invention relates to the field of audio processing technology, and specifically to a method for refining musical instrument timbre conversion based on time-frequency characteristics. Background Technology

[0002] Instrument timbre conversion allows for more diverse and personalized musical performances, while also reducing human and material costs and improving music production efficiency. Therefore, efficiently converting instrument timbres, and generating various transitional timbres during the conversion process, is of great significance for creating timbre sample libraries and real-time performance.

[0003] Instrument timbre conversion is an important field in music synthesis. Electronic instruments and synthesizers in music synthesis can simulate the timbres of various instruments and produce high-quality music audio. However, it is difficult to simulate the resonance of the instrument, the performer's technique, and other factors that produce the authentic instrument timbre. Instrument timbre conversion can preserve the rhythm, melody, performer's style, and emotional expression of the original music.

[0004] Most current timbre conversion models require training on both source and target audio data to derive conversion rules. These rules then convert the acoustic features of the source instrument signal to the acoustic features of the target instrument, synthesizing the final audio. Such models typically require extensive training with large amounts of audio data and consume significant time and computational resources. This can be limiting for niche instruments or those with limited data, and in extreme cases, the timbre conversion is uncontrollable; it can only convert the timbre of the source instrument to the target instrument's timbre, failing to produce a transitional timbre between the two. A timbre conversion method that does not require model training is based on digital filter models. However, this method requires modeling the frequency response of different instruments and has a narrower applicability.

[0005] Therefore, exploring efficient instrument timbre conversion methods that do not require model training, and that can controllably and gradually convert one instrument timbre into another, while also producing a transitional timbre between the two, is of great significance for real-time performance and the creation of timbre sample libraries. Summary of the Invention

[0006] The purpose of this invention is to overcome the above-mentioned defects in the prior art and provide a method for instrument timbre conversion based on refined time-frequency characteristics.

[0007] The objective of this invention can be achieved by adopting the following technical solutions:

[0008] A method for refining musical instrument timbre based on time-frequency features, the method comprising the following steps:

[0009] S1. Extract timbre features from the instrument audio dataset and establish a target timbre feature library; among which, timbre features include fundamental frequency, fundamental pulse marker sequence, duration, temporal bilateral peak envelope, harmonic energy ratio, and fractional harmonic energy ratio;

[0010] S2. Based on the target instrument type, obtain the time-domain features of the target instrument timbre from the target timbre feature library. Perform time-domain feature transformation on the fundamental frequency, duration, and time-domain amplitude envelope of the audio to be transformed by adjusting the fundamental frequency, modifying the duration, modulating the time-domain amplitude, and interpolating the fundamental frequency. This will yield the single-tone audio y(n) after time-domain feature transformation.

[0011] S3. Frame and window the monotone audio y(n), extract the harmonic energy ratio and the energy ratio of each fractional harmonic. Obtain the frequency domain features of the target instrument's timbre from the target timbre feature library according to the target instrument type. Perform frequency domain feature transformation by adjusting the harmonic energy ratio and the energy ratio of each fractional harmonic in each frame of the monotone audio y(n). Then, splice the audio frames to obtain the monotone audio output(n) with the timbre of the target instrument.

[0012] S4. Repeat steps S2 and S3 for all single-note audio frequencies of the target instrument to complete the timbre conversion.

[0013] Furthermore, the musical instrument audio dataset refers to a dataset containing single-note audios of different instruments at different pitches. Because different instruments and different pitches have different timbres, the audio in this dataset can provide time-frequency features of different timbres for the subsequent establishment of a target timbre feature library.

[0014] Furthermore, the target timbre feature library is established by extracting the time-frequency features of each single-tone audio from the instrument audio dataset. Each single-tone audio from each instrument has a corresponding fundamental frequency, duration, fundamental impulse marker sequence, time-domain bilateral peak envelope, harmonic energy ratio, and fractional harmonic energy ratio. The harmonic energy ratio and fractional harmonic energy ratio are obtained by windowing and segmenting the single-tone audio into frames, resulting in the harmonic energy ratio and fractional harmonic energy ratio of each frame. This target timbre feature library contains the time-frequency features of different instruments and different pitches, providing the required time-frequency features for instrument timbre conversion.

[0015] Furthermore, the fractional harmonic energy is defined as follows: when the frequency of the m-th harmonic in a certain frame of a single-tone audio is f... m Frequency (f) m-1 +β(f m -f m-1The frequency of the fractional harmonic (m-1+β) is expressed as the frequency of the fractional harmonic, where m = 1, 2, ..., M. f f0 = 0, β is a fraction, and β ranges from 0 to 1, M f For the harmonic order, the frequency is (f m-1 +β(f m -f m-1 The fractional harmonics of frequency f are collectively referred to as beta fractional harmonics. m-1 +β(f m -f m-1 The energy proportions of fractional harmonics are collectively referred to as the β fractional harmonic energy proportion. This step is taken because fractional harmonics have a significant impact on the timbre of musical instruments, and using the fractional harmonic energy proportion as a characteristic can better characterize the timbre of musical instruments.

[0016] Furthermore, step S1 is performed as follows:

[0017] S101. Use a short-time energy-based dual-threshold endpoint detection method to remove the silent segments of single-tone audio from the dataset and obtain the duration feature, where the duration is the time length of the non-silent segment of the single-tone audio.

[0018] S102. The fundamental frequency features of a single-tone audio are extracted using narrowband spectral energy frequency estimation based on time-domain autocorrelation and iterative resampling. First, a coarse estimate of the fundamental frequency of the instrument's single-tone audio is obtained through time-domain autocorrelation. Then, the coarse estimate of the fundamental frequency is gradually approximated to the true fundamental frequency of the signal through iterative narrowband spectral energy frequency estimation. Finally, a fine estimate of the fundamental frequency of the single-tone audio is obtained. The process is as follows:

[0019] S102.1: First, the single-tone audio is divided into frames. One frame is then subjected to time-domain autocorrelation to obtain a coarse estimate of the fundamental frequency, denoted as f. 0_start Assume the audio signal sampling rate is Fs;

[0020] S102.2: According to f 0_start Determine the resampling rate f s_new First, determine the value of integer c according to formula (1);

[0021] f 0_start ×2 c ≤Fs≤f 0_start ×2 c+1 Formula (1)

[0022] Comparison |Fs-f 0_start ×2 c+1 |and|Fs-f 0_start ×2 c The magnitude of |Fs-f| 0_start ×2 c+1 |≤|Fs-f 0_start×2 c |,f s The calculation formula is as follows:

[0023] f s =f 0_start ×2 c+1 Formula (2)

[0024] When |Fs-f 0_start ×2 c+1 |≥|Fs-f 0_start ×2 c |time

[0025] f s =f 0_start ×2 c Formula (3)

[0026] S102.3: Use the determined resampling rate f s The frame is resampled using cubic spline interpolation; then, N1 points of the frame are windowed and subjected to discrete Fourier transform to obtain its amplitude spectrum F(k).

[0027] S102.4: Find the maximum peak value of the amplitude spectrum F(k), and the peak value index is k0;

[0028] S102.5: Based on a narrow band formed by the amplitude spectrum around points before and after k0, and referring to existing frequency estimation formulas based on narrow band spectral energy, the frequency f′ of the audio signal is estimated:

[0029]

[0030] S102.6: According to f 0_start Determine the harmonic order value of f′:

[0031]

[0032] Where round() is the rounding function, and m indicates that f′ is the mth harmonic;

[0033] S102.7: Obtain the new baseband f 0_new The fundamental frequency at this time is

[0034]

[0035] S102.8: Iteratively estimate the fundamental frequency; set a threshold α, and when the difference between the output fundamental frequencies of two adjacent iterations is greater than the threshold α, i.e., |f 0_new -f 0_start |>α, with f 0_start =f 0_newRepeat steps S102.2 to S102.7; when the difference between the fundamental frequencies of two adjacent iterations is less than the threshold α, i.e., |f 0_new -f 0_start |≤α,f 0_new Then the fundamental frequency of the infinitely approximate audio signal, f 0_new This is the precise estimate of the fundamental frequency of a single-tone audio frequency, denoted as f. 0_end The smaller the threshold α, the closer the estimated frequency is to the true fundamental frequency of the audio signal.

[0036] This step selects a single frame of a monotone audio signal for fundamental frequency estimation because the fundamental frequency variation is small for monotone audio signals from musical instruments. Therefore, the fundamental frequency of a single frame can be used to characterize the fundamental frequency of the monotone audio signal. Existing fundamental frequency estimation methods, such as short-time autocorrelation, cepstral methods, and gene trajectory tracing, all suffer from inaccurate fundamental frequency estimation. This proposed method can effectively solve the problem of inaccurate fundamental frequency estimation in musical instrument audio signals.

[0037] S103: Perform pitch pulse tagging on the single-tone audio and obtain the pitch pulse tagging sequence. The process is as follows:

[0038] S103.1: The period of a single tone is denoted as T0 = Fs / f 0_end On the time-domain waveform of a single-tone audio signal, determine the positions of the maximum peak points for three consecutive cycles of the non-silent segment, namely (0~T0), (T0~2T0), and (2T0~3T0), denoted as M1, M2, and M3, where M... λ This indicates the position of the marker point in the λth period; first, let λ = 3, and then perform the following steps;

[0039] S103.2: According to M λ-2 M λ-1 M λ Determine the number of sampling points T for the fundamental period λ-2 T λ-1 The difference d between the number of sampling points in two adjacent periods λ ;

[0040] T λ-2 =M λ-1 -Mλ-2 formula (7)

[0041] T λ-1 =M λ -M λ-1 Formula (8)

[0042] d λ =T λ-1 -T λ-2 Formula (9)

[0043] S103.3: Predict the position of the next synchronization marker

[0044] M λ+1 =M λ +T λ-1 +d λ Formula (10)

[0045] Find the local maximum of a single tone near that location. The location of the local maximum is marked as point M. λ+1 ; Calculate the number of sampling points T in the λth period. λ :

[0046] T λ =M λ+1 -M λ Formula (11)

[0047] Continue executing steps S103.2 to S103.3 to find the next marker point. When the predicted marker point position exceeds the duration of a single tone, the marking ends, and the pitch pulse marker sequence is obtained, denoted as T. N = [T1, T2, ..., T M ], where M is the number of periods; this step predicts the position of the next marker point by using the positions of the marker points in the first two periods. Considering the periodic change of the fundamental frequency of the audio, a better fundamental pulse marking effect can be obtained;

[0048] S104. A sliding window-based peak detection method is used to obtain the temporal bilateral peak envelope features of a single-tone audio. The process is as follows: Define a window with a length equal to the temporal period length N2 of the single-tone audio, and a moving length of 2N2 / 3. Find the maximum and minimum points of each window, and then obtain the temporal bilateral peak envelope of the single-tone audio through spline fitting. This step, using a sliding window-based peak detection method, can extract individual transient features, more accurately reflecting the instantaneous envelope changes of the single-tone audio, and solving the problem of ignoring transient feature extraction in the temporal envelope acquisition method.

[0049] S105. Resample the single-tone audio, divide it into frames, and determine the harmonic frequency sequence of each frame. The process is as follows: Window each frame of single-tone audio, obtain the amplitude spectrum through Fourier transform, and determine the harmonic frequency sequence of each frame based on the fundamental frequency f of the single-tone audio. 0_end At a frequency of mf 0_end Find the frequency corresponding to the peak point of the amplitude spectrum nearby, and take it as the frequency f of the m-th harmonic. m ; make m = 1, 2, ..., M f A total of M were obtained f The frequency sequence of the second harmonic is denoted as f = [f1, f2, ..., f...]. Mf This step involves resampling the single-tone audio at a sampling rate of f. 0_endWhen the frequency is an integer multiple of 2 and a power of 2, the frequency components in the signal spectrum are concentrated on the fundamental frequency and harmonics, thus obtaining more accurate spectrum analysis results.

[0050] S106. Based on the harmonic frequency sequence of each frame, calculate the proportion of each harmonic energy in each frame. The process is as follows: First, based on the harmonic frequency sequence of each frame, using the harmonic frequency f... m The bandwidth can be taken as the fundamental frequency f, where the center frequency is the m-th harmonic. 0_end 10%, that is, |f m -0.05f 0_end f m +0.05f 0_end The spectral energy within the frequency range is used as harmonic f m The energy of each harmonic is calculated to obtain the harmonic energy ratio of each frame.

[0051] S107. Based on the harmonic frequency sequence of each frame, calculate the β fractional harmonic frequency sequence of each frame, and obtain the β fractional harmonic energy ratio of each frame. The process is as follows: using the frequencies f of the β fractional harmonic frequency sequence... β_ε The bandwidth is taken as the fundamental frequency f, which is the center frequency. 0_end 10%, that is, |f β_ε -0.05f 0_end f β_ε +0.05f 0_end |The spectral energy within the frequency range is used as f β_ε The energy, where f β_ε The ∈ represents the ∈ ith frequency of the β fractional harmonic frequency sequence. By calculating the harmonic energy of each frequency of the β fractional harmonic, the proportion of β fractional harmonic energy in each frame is obtained. The values ​​of β are 1 / 2, 1 / 4, 3 / 4, 1 / 8 and 7 / 8.

[0052] S108. Obtain the timbre features of each single-note audio in the instrument audio dataset through the above steps, and establish a target timbre feature library.

[0053] Furthermore, step S2 is as follows:

[0054] S201. Adjust the fundamental frequency of the audio to be converted. The process is as follows: Repeat step S102 to obtain the fundamental frequency f of the audio to be converted. 0_end Based on the fundamental frequency characteristics f of the target instrument 0_target The fundamental frequency of the audio to be converted is adjusted by interpolation to obtain the single-tone audio x1(n) after fundamental frequency adjustment. x1(n) is an audio with transitional timbre. Cubic spline interpolation is used as the interpolation method.

[0055] S202. Adjust the duration of the single-tone audio x1(n) by deleting or inserting the fundamental pulse marker unit. The process is as follows: Repeat the duration acquisition method in step S101 to obtain the duration of the single-tone audio x1(n), denoted as t0, and then adjust the duration according to the target fundamental frequency f. 0_target The single-tone audio x1(n) is marked with a fundamental pulse, and windowing is applied to each marking period to obtain a short-time frame sequence of the fundamental tone; based on the duration t0 of the single-tone audio x1(n) and the duration t of the target instrument... target Calculate the pitch length modification ratio, delete and insert new pitch short frames to complete the pitch length modification of the single-tone audio x1(n), and obtain the single-tone audio x2(n), which is an audio with transitional timbre; this step uses a window function for smoothing, which can avoid discontinuity problems at the periodic boundary;

[0056] S203, According to the target base frequency f 0_target The single-tone audio x2(n) is marked with a fundamental pulse, and amplitude modulation is performed on each marked period to achieve temporal envelope adjustment of the single-tone audio x2(n). The process is as follows: First, the upper and lower envelope peak values ​​of each marked period are calculated. Then, the scaling factor is calculated based on the temporal peak envelope characteristics of the target instrument to achieve upper and lower envelope amplitude modulation. Next, the waveforms of each period after amplitude modulation are spliced ​​together. Finally, a Gaussian filter is used for smoothing to obtain the single-tone audio x3(n) with pitch adjustment. x3(n) is an audio with transitional timbre. The smoothing process in this step is to avoid abrupt changes between different periods and reduce the discontinuity caused by the amplitude scaling factor, making the peak envelope adjustment of the single-tone audio x3(n) smoother.

[0057] S204. Based on the target fundamental pulse marker sequence, perform fundamental frequency synchronization interpolation on the single-tone audio x3(n), as follows: Based on the target fundamental frequency f 0_target The single-tone audio x3(n) is marked with a fundamental pulse. Then, based on the fundamental pulse mark sequence characteristics of the target instrument, the number of sampling points for each fundamental period of the target instrument is obtained. Interpolation is used to make the number of sampling points for each period of x3(n) consistent with the number of sampling points for each period of the target instrument; this is called fundamental-tone synchronous interpolation. The resulting single-tone audio y(n) is an audio with a transitional timbre. Cubic spline interpolation is used as the interpolation method. This fundamental-tone synchronous interpolation step makes the single-tone audio y(n) more consistent with the fundamental pitch jitter of the target instrument, solving the problem of unstable instrument pitch that may be ignored in existing timbre conversion methods.

[0058] Furthermore, step S3 is as follows:

[0059] S301. Resample and frame-window the single-tone audio y(n) to obtain the harmonic frequency sequence of each frame. The resampling method uses cubic spline interpolation, and the process is as follows: Based on the target fundamental frequency f0_target Determine the resampling rate, resample the single-tone audio y(n), then frame and window it; denote the time-domain signal of the i-th frame of the single-tone audio y(n) as y i (n), i = 1, 2, ..., K y K y For the frame number, get y i (n) is a harmonic frequency sequence; this step resamples the single-tone audio, enabling y to... i (n) After the Fast Fourier Transform, the frequency components on the spectrum are concentrated on the fundamental frequency and harmonics, which can more accurately perform frequency domain feature conversion.

[0060] S302, Calculate y i (n) Harmonic energy acquisition ratio, y acquisition i The proportion of harmonic energy of (n) is denoted as o. i ;

[0061] S303, Extract y i The time-domain waveforms of each harmonic of (n) are obtained as follows: A window-function-based FIR bandpass filter is used to obtain y. i (n) The time-domain signals of each harmonic, y i The time-domain signal of the j-th harmonic of (n) is denoted as y. ij (n); This step uses a window-function-based FIR bandpass filter because FIR filters have a stable linear phase and the window function method is fast and time-efficient for implementing high-order filters;

[0062] S304, Based on the harmonic energy ratio v of the i-th frame signal of the target musical instrument i Adjust y i (n) The proportions of each harmonic energy, the process is as follows:

[0063] y ij (n) Perform a Fourier transform to obtain the spectrum Y ij (k), based on the harmonic energy ratio v of the i-th frame of the target instrument. i y obtained from S302 i The harmonic energy ratio of (n) is oi; v i The j-th coefficient is denoted as v ij , will o i The j-th coefficient is denoted as o ij The harmonic energy ratio adjustment coefficient γ is calculated according to formula (12). ij :

[0064]

[0065] Where, γ ij Adjusting the energy ratio γ iThe energy adjustment coefficient of the j-th harmonic is used to adjust the energy of each harmonic according to formula (13):

[0066]

[0067] Among them, Z ij (k) is the spectrum of the j-th harmonic in the i-th frame after energy adjustment; according to formula (14), the spectra of each harmonic in the i-th frame are superimposed to obtain the harmonic spectrum Z. i (k), Z i (k) is a spectrum with transitional timbres;

[0068]

[0069] This step is based on the harmonic energy ratio method to perform frequency domain feature conversion, because harmonic energy can more stably characterize the timbre of an instrument, is not affected by amplitude changes, provides more stable timbre information and better describes the relative intensity between harmonic components;

[0070] S305, Based on the β fractional harmonic energy ratio of the i-th frame signal of the target musical instrument. i_β Adjust y i The proportion of the β fractional harmonic energy of (n) is as follows:

[0071] S305.1: According to y i The harmonic frequency sequence of (n) determines y i The frequency sequence of the β fractional harmonic (n) is used to obtain the energy ratio of the β fractional harmonic and the time-domain signal of each β fractional harmonic. The process is as follows: In the part of obtaining the energy ratio of the β fractional harmonic, y is obtained. i The proportion of the β fractional harmonic energy of (n) i_β In each time-domain signal portion of the β fractional harmonic, an FIR bandpass filter is used to filter out the y-values. i The time-domain signals of each frequency of the β fractional harmonic in (n) are denoted as y. ij_β (n);

[0072] S305.2: Based on the β fractional harmonic energy ratio v of the target instrument in the i-th frame i_β Adjust y i The proportion of β-fractional harmonic energy in (n) is as follows:

[0073] v i_β The j-th coefficient is denoted as v ij_β , will o i_β The j-th coefficient is denoted as o ij_β The energy adjustment coefficient γ of the β fractional harmonic is calculated according to formula (15). ij_β :

[0074]

[0075] y ij_β (n) Perform a Fourier transform to obtain the spectrum. Adjust the energy of the β fractional harmonic according to formula (16):

[0076]

[0077] Where, γ ij_β Adjusting the energy ratio γ i_β The j-th energy adjustment coefficient; Z ij_β (k) is the j-th spectrum of the β fractional harmonic of the i-th frame after energy adjustment;

[0078] By superimposing the spectra of the β fractional harmonics according to formula (17), the spectrum Z containing only the β fractional harmonics is obtained. i_β (k);

[0079]

[0080] S305.3: For each fraction where β is greater than 0 and less than 1, repeat steps S305.1 to S305.2 to obtain the spectrum of each fractional harmonic;

[0081] S305.4: According to formula (18), the spectrum Z of each fractional harmonic is... i_β (k) and harmonic spectrum Z i (k) are superimposed;

[0082]

[0083] This step of adjusting the fractional harmonic energy takes into account that fractional harmonics have a certain impact on the timbre of musical instruments;

[0084] S305.5: Output the frequency domain of the i-th frame. i (k) Fourier transform to the time domain to obtain the i-th frame time domain signal output after frequency domain feature transformation. i (n);

[0085] S306, K of the single-tone audio value y(n) y Steps S302 to S305 are repeated for each frame to obtain the time-domain signal output after frequency domain feature transformation of each frame. i (n), output i (n) is concatenated to obtain a single-tone audio output(n) with the timbre of the target instrument.

[0086] The present invention has the following advantages and effects compared with the prior art:

[0087] 1) This invention converts the audio to be converted into audio with the timbre of the target instrument through digital signal processing. Compared with existing machine learning timbre conversion methods, this invention does not require a large amount of sample data for training and does not require a large amount of computing resources and time. It solves the problem that niche instruments or instruments with a small amount of sample data are difficult to convert due to insufficient data.

[0088] 2) This invention is based on time-frequency features for conversion. During the time-frequency feature conversion process, various transitional timbres between the timbres of the audio to be converted and the timbres of the target instrument can be obtained. For example, after time-domain amplitude modulation, transitional timbres x2(n) can be obtained, which solves the problem that it is difficult to obtain transitional timbres in existing timbre conversion methods or music synthesizers.

[0089] 4) In the fundamental frequency extraction feature part, this invention adopts a narrowband spectral energy frequency estimation method based on time-domain autocorrelation and iterative resampling. Compared with the existing fundamental frequency estimation methods, this fundamental frequency estimation method can effectively solve the problem of inaccurate fundamental frequency estimation in instrument audio signals. The time-domain feature transformation method of pitch synchronization interpolation is used to make y4(n) more consistent with the pitch jitter of the target instrument, which solves the problem of unstable instrument pitch that may be ignored in the existing timbre conversion methods.

[0090] 5) This invention resamples the single-tone audio before frequency domain feature transformation, with a sampling rate of the target fundamental frequency f. 0_target An integer multiple of y, and when y is a power of 2, can make y i (n) After the Fast Fourier Transform, the frequency components on the spectrum are concentrated on the fundamental frequency and harmonics, thus obtaining more accurate harmonic energy and fractional harmonic energy; the frequency domain feature conversion is performed by the method based on the ratio of harmonic energy and the ratio of fractional harmonic energy, which is not affected by amplitude changes, provides more stable timbre information and relative intensity between harmonic components, and can better characterize the timbre of the instrument and realize the timbre conversion of the instrument. Attached Figure Description

[0091] The accompanying drawings, which are included to provide a further understanding of the invention and form part of this application, illustrate exemplary embodiments of the invention and, together with their description, serve to explain the invention and do not constitute an undue limitation thereof. In the drawings:

[0092] Figure 1 This is a schematic diagram of the musical instrument timbre conversion method based on time-frequency features disclosed in this invention;

[0093] Figure 2 This is a schematic diagram of fractional harmonics in an embodiment of the present invention;

[0094] Figure 3 This is a diagram of the fundamental pulse marking in an embodiment of the present invention;

[0095] Figure 4 This is a diagram showing the harmonic frequencies and their corresponding harmonic energy ratios in the first frame of this invention.

[0096] Figure 5 These are waveforms before and after time-domain amplitude envelope adjustment in an embodiment of the present invention; wherein, Figure 5 (a) is the waveform before time-domain amplitude envelope adjustment; Figure 5 (b) is the waveform after time-domain amplitude envelope adjustment;

[0097] Figure 6 This is the time-domain waveform and frequency-domain logarithmic amplitude spectrum of the first frame in this embodiment of the invention, which contains only harmonics; wherein, Figure 6 (a) is a time-domain waveform diagram containing harmonics; Figure 6 (b) is the frequency domain logarithmic amplitude spectrum including harmonics;

[0098] Figure 7 This is the frequency domain logarithmic amplitude spectrum of the first frame in this embodiment of the invention, in which 1 / 2, 1 / 4, and 3 / 4 fractional harmonics are added sequentially.

[0099] Figure 8 This is a schematic diagram of the audio time-domain waveform with the target timbre in an embodiment of the present invention. Detailed Implementation

[0100] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0101] Example 1

[0102] This implementation case discloses a method for refined musical instrument timbre conversion based on time-frequency characteristics, such as... Figure 1 As shown, the specific steps are as follows:

[0103] S1. Extract timbre features from the instrument audio dataset and establish a target timbre feature library; among which, timbre features include fundamental frequency, fundamental pulse marker sequence, duration, temporal bilateral peak envelope, harmonic energy ratio, and fractional harmonic energy ratio;

[0104] The instrument audio dataset refers to a dataset containing single-note audio from different instruments at different pitches. The instrument audio dataset used in this implementation example consists of single-note audio from real instruments, all with a sampling rate of 44.1kHz, a precision of 16bit, and formats of wav, aif, and mp3. Specific audio samples are shown in Table 1.

[0105] Table 1. Musical Instrument Audio Dataset

[0106] Musical Instrument Types Pitch range Number of single notes piano A0-C8 88 violin G3-G7 48 xylophone C3-F6 42 small E3-Eb6 36 flute B3-Bb6 36

[0107] The target timbre feature library is established by extracting the timbre features of each single-tone audio from the instrument audio dataset. Each single-tone audio of each instrument has a corresponding fundamental frequency, duration, fundamental pulse marker sequence, temporal bilateral peak envelope, harmonic energy ratio, and fractional harmonic energy ratio. The harmonic energy ratio and fractional harmonic energy ratio refer to the harmonic energy ratio and fractional harmonic energy ratio of each frame obtained after windowing and segmenting the single-tone audio.

[0108] The fractional harmonic energy is defined as follows: when the frequency of the m-th harmonic in a frame of a single-tone audio is f... m Frequency (f) m-1 +β(f m -f m-1 The frequency of the fractional harmonic (m-1+β) is expressed as the frequency of the fractional harmonic, where m = 1, 2, ..., M. f f0 = 0, β is a fraction, and β ranges from 0 to 1, M f This refers to the harmonic order. For example... Figure 2 The diagram shown is a fractional harmonic diagram, with the frequency (f) m-1 +β(f m -f m-1 The fractional harmonics of frequency f are collectively referred to as beta fractional harmonics. m-1 +β(f m -f m-1 The energy proportions of fractional harmonics are collectively referred to as the β fractional harmonic energy proportions.

[0109] S101. Use a short-time energy-based dual-threshold endpoint detection method to remove the silent segments of single-tone audio from the dataset and obtain the duration feature, where the duration is the time length of the non-silent segment of the single-tone audio.

[0110] S102. The fundamental frequency features of a single-tone audio are extracted by narrowband spectral energy frequency estimation based on time-domain autocorrelation and iterative resampling. First, the coarse estimate of the fundamental frequency of the single-tone audio of the instrument is obtained by the time-domain autocorrelation method. Then, the coarse estimate of the fundamental frequency is gradually approximated to the true fundamental frequency of the signal by iterative narrowband spectral energy frequency estimation. Finally, the fine estimate of the fundamental frequency of the single-tone audio is obtained.

[0111] This embodiment uses a Blackman window with fast sidelobe attenuation. Since the main lobe energy is concentrated, the main lobe energy can approximate the energy of the entire spectrum. α is set to 0.001, and N1 is set to 4096. For the single-tone violin A3, the fundamental frequency is coarsely estimated using autocorrelation to be 218.531Hz, and a finely estimated fundamental frequency of 220.3234Hz is obtained through iterative narrowband spectrum energy frequency estimation.

[0112] S103: Mark the pitch pulses on the single-tone audio and obtain the pitch pulse mark sequence;

[0113] Because the fundamental frequency of violin A3 is 220.3234Hz, the number of sampling points for one cycle T0 of violin A3 is approximately 200. The positions of the maximum peak points for three consecutive cycles in the non-silent section of violin A3 are determined using T0: M1 = 1134, M2 = 1332, M3 = 1533. Based on formulas (7), (8), and (9), the number of fundamental frequency sampling points T1 = 198, T2 = 201, and the difference in sampling points between two adjacent cycles d3 = T2 - T1 = 3 are determined. The position of the next synchronization marker is predicted as M4 = M3 + T2 + d3 = 1737. The maximum value near this position is found, and the position corresponding to the maximum value is the marker point. The search continues for the next marker point. The fundamental frequency pulse marker sequence for single-tone violin A3 is T... N = [198, 201, ..., 203, 200]. Figure 3 This is a pitch pulse marker diagram for the periodic single-tone audio portion of this embodiment. This method can accurately locate the marker points for each period and obtain the pitch pulse marker sequence.

[0114] S104. A peak detection method based on a sliding window is used to obtain the temporal bilateral peak envelope features of a single-tone audio.

[0115] S105. Resample the single-tone audio, divide it into frames, and determine the harmonic frequency sequence of each frame.

[0116] S106. Based on the harmonic frequency sequence of each frame, calculate the proportion of each harmonic energy in each frame.

[0117] For violin A3 in this embodiment, f 0_start =220.3234Hz. According to step S102.2, the resampling rate is determined to be 56403Hz. Then, the frame is divided and windowed. The frame length is 16384 sampling points, and the frame shift is 1 / 2 of the frame length. The single-tone audio violin A3 is divided into 3 frames, and the Hamming window function is used.

[0118] The harmonic energy calculated in this embodiment is for harmonics that just exceed 10000Hz. For a violin A3 with a fundamental frequency of 220.3234Hz, the harmonic order M that just exceeds 10000Hz is... f =45. The harmonic frequencies of the first frame of violin A3 are [224.54, 443.91, 665.01, ..., 10174.52], and the corresponding harmonic energy ratios are [0.0739, 1, 0.1030, ..., 1.4232e-06]. For example... Figure 4 The figure shows the harmonic energy ratio corresponding to the harmonic frequencies of the first frame of violin A3.

[0119] S107. Based on the harmonic frequency sequence of each frame, calculate the β fractional harmonic frequency sequence of each frame and obtain the β fractional harmonic energy ratio of each frame.

[0120] The fractional harmonic energies of the first frame of violin A3 are as follows: When β = 1 / 2, the frequency sequence of the 1 / 2 fractional harmonics is [112.27, 333.36, 554.46, ..., 10034.27], and the energy ratio of the 1 / 2 fractional harmonics is [1, 0.1582, 0.1741, ..., 6.5411e-04]; when β = 1 / 4, the frequency sequence of the 1 / 4 fractional harmonics is [55.27, 278.09, 499.18, ..., 9963.40], and the energy ratio of the 1 / 4 fractional harmonics is [1, 0.0191, 0.0096, ..., 7.505e-05]; when β = 3 / 4, the frequency sequence of the 3 / 4 fractional harmonics is [167.54, 388.64, 609.73]. When β = 1 / 8, the frequency sequence of the 1 / 8 fractional harmonics is [27.63, 250.07, 471.55, ..., 9928.47], and the energy ratio of the 1 / 8 fractional harmonics is [1, 0.0044, 0.0035, ..., 1.0264e-05]; when β = 7 / 8, the frequency sequence of the 7 / 8 fractional harmonics is [195.18, 416.278, 637.37, ..., 10137.1], and the energy ratio of the 7 / 8 fractional harmonics is [0.059, 0.0526, 1, ..., 4.147e-04].

[0121] Using the above method, the harmonic energy ratio and the characteristics of each fractional harmonic energy ratio of the other two frames in violin A3 are extracted to complete the time-frequency feature extraction of violin A3.

[0122] S108. Obtain the timbre features of each single-note audio in the instrument audio dataset through the above steps, and establish a target timbre feature library.

[0123] S2. Based on the target instrument type, obtain the time-domain features of the target instrument timbre from the target timbre feature library. Perform time-domain feature transformation on the fundamental frequency, duration, and time-domain amplitude envelope of the audio to be transformed by adjusting the fundamental frequency, modifying the duration, modulating the time-domain amplitude, and interpolating the fundamental frequency. This will yield the single-tone audio y(n) after time-domain feature transformation.

[0124] S201. Adjust the base frequency of the audio to be converted;

[0125] In this embodiment, the timbre of the target instrument, violin A3, is obtained as the target timbre. The audio to be converted is a single note from piano key A4, with a fundamental frequency of 440.0805Hz. The fundamental frequency f of the target instrument, violin A3, is... 0_target The frequency is 220.3234Hz. Obtain the time-domain signal x1(n) after adjusting the fundamental frequency.

[0126] S202, Adjust the duration of the single-tone audio value x1(n) by deleting or inserting the pitch pulse marker unit;

[0127] Get the duration t of the target instrument, violin A3. target Given a pitch length of 0.92s and a pitch length of t0 of x1(n) of 1.48s, calculate the pitch length modification ratio η = t target / t0 = 0.62. When η < 1, shorten x1(n); when η > 1, lengthen x1(n). It is not recommended to lengthen x1(n) when η > 2.

[0128] Calculate when η < 1 and This means that len ​​pitch short frames need to be deleted from every 10 pitch short frames. When len is not an integer, it can be rounded up or down, so that the obtained x2(n) is approximately the same as the pitch of the target instrument.

[0129] S203, According to the target base frequency f 0_target A pitch pulse is marked on the single-tone audio x2(n), and amplitude modulation is performed on each marking period to achieve temporal envelope adjustment of the single-tone audio x2(n), such as... Figure 5 These are the waveforms before and after the time-domain amplitude envelope adjustment of x2(n) in this embodiment. The transition timbre x3(n) is obtained after the amplitude adjustment.

[0130] S204. Based on the target pitch pulse marker sequence, perform pitch synchronization interpolation on the single-tone audio x3(n);

[0131] S3. Frame and window the monotone audio y(n), extract the harmonic energy ratio and the energy ratio of each fractional harmonic. Obtain the frequency domain features of the target instrument's timbre from the target timbre feature library according to the target instrument type. Perform frequency domain feature transformation by adjusting the harmonic energy ratio and the energy ratio of each fractional harmonic in each frame of the monotone audio y(n). Then, splice the audio frames to obtain the monotone audio output(n) with the timbre of the target instrument.

[0132] S301. Resample, frame, and window the single-tone audio y(n) to obtain the harmonic frequency sequence of each frame of the signal;

[0133] S302, Calculate y i (n) Harmonic energy acquisition ratio, y acquisition i The proportion of harmonic energy of (n) is denoted as o. i ;

[0134] Referring to step S105, which involves resampling, framing, and windowing, the monotone audio after temporal feature conversion in this embodiment is resampled, framed, and windowed. In this embodiment, the fundamental frequency of the audio to be converted, y(n), is 220.0805Hz, the source sampling rate is 44.1kHz, and the resampling rate is determined to be 56320Hz. After resampling, based on the frame length and frame shift, y(n) can be divided into 3 frames, with a frame length of 16384 sampling points and a frame shift of 1 / 2 of the frame length.

[0135] In this embodiment, the harmonic frequency sequence of the first frame y1(n) of y(n) is denoted as:

[0136] f=[223.75, 443.75, 663.75,..., 10170.75],

[0137] The harmonic energy ratio of the first frame is denoted as o1 = [0.3391, 1, 0.1388, ..., 2.3585e-08].

[0138] S303, Extract y i Time-domain waveforms of each harmonic of (n);

[0139] When using an FIR bandpass filter to obtain harmonic time-domain signals, the harmonic frequency f m The center frequency is f, and the passband bandwidth is the target fundamental frequency f. 0_target 0.2 times, that is, B = 0.2f 0_target The time-domain signals of each harmonic are extracted. In this embodiment, the harmonic frequency sequence of the first frame y1(n) of y(n) is denoted as f = [223.75, 443.75, 663.75, ..., 10170.75]. 0_target=220.3234Hz, so B=44; Using the frequencies of f as the center frequency and the passband bandwidth as 44Hz, extract the time domain signals of each harmonic of the first frame.

[0140] S304, Based on the harmonic energy ratio v of the i-th frame signal of the target musical instrument i Adjust y i (n) The proportions of each harmonic energy, the process is as follows:

[0141] y ij (n) Perform a Fourier transform to obtain the spectrum Y ij (k), based on the harmonic energy ratio v of the i-th frame of the target instrument. i y obtained from S302 i The proportion of harmonic energy of (n) i ; will v i The j-th coefficient is denoted as v ij , will o i The j-th coefficient is denoted as o ij The harmonic energy ratio adjustment coefficient γ is calculated according to formula (12). ij .

[0142] For y1(n), o1=[0.3391, 1, 0.1388, ..., 2.3585e-08], v1=[0.0739, 1, 0.1030, ..., 1.4232e-06], so γ1=[0.2181, 1, 0.7419, ..., 60.34].

[0143] Adjust the energy of each harmonic according to formula (13); superimpose the spectra of each harmonic in the i-th frame according to formula (14) to obtain the harmonic spectrum Z. i (k), Z i (k) is the spectrum with transitional timbre. Adjusting the harmonic energy ratio of y1(n) yields the spectrum Z1(k), as shown below. Figure 6 In this embodiment, the first frame contains only the time-domain waveform and frequency-domain logarithmic amplitude spectrum of harmonics.

[0144] S305, Based on the β fractional harmonic energy ratio of the i-th frame signal of the target musical instrument. i_β Adjust y i The proportion of β-fractional harmonic energy of (n):

[0145] S305.1: According to y i The harmonic frequency sequence of (n) determines y i The frequency sequence of the β fractional harmonic (n) is used to obtain the energy ratio of the β fractional harmonic and the time-domain signal of each β fractional harmonic. The process is as follows: In the part of obtaining the energy ratio of the β fractional harmonic, y is obtained. iThe proportion of the β fractional harmonic energy of (n) i_β In each time-domain signal portion of the β fractional harmonic, an FIR bandpass filter is used to filter out the y-values. i The time-domain signals of each frequency of the β fractional harmonic in (n) are denoted as y. ij_β (n);

[0146] S305.2: Based on the β fractional harmonic energy ratio v of the target instrument in the i-th frame i_β Adjust y i The proportion of β-fractional harmonic energy in (n) is as follows:

[0147] v i_β The j-th coefficient is denoted as v ij_β , will o i_β The j-th coefficient is denoted as O ij_β The energy adjustment coefficient γ of the β fractional harmonic is calculated according to formula (15). ij_β :

[0148] y ij_β (n) Perform a Fourier transform to obtain the spectrum. Adjust the energy of the β fractional harmonic according to formula (16):

[0149] By superimposing the spectra of the β fractional harmonics according to formula (17), the spectrum Z containing only the β fractional harmonics is obtained. i_β (k);

[0150] S305.3: For each fraction where β is greater than 0 and less than 1, repeat steps S305.1 to S305.2 to obtain the spectrum of each fractional harmonic;

[0151] S305.4: According to formula (13), the spectrum Z of each fractional harmonic is... i_β (k) and harmonic spectrum Z i (k) are superimposed;

[0152] Based on the harmonic frequencies of y1(n) [223.75, 443.75, 663.75, ..., 10170.75], determine the frequency sequence of the 1 / 2th fractional harmonics, denoted as f. 1_1 / 2 =[111.87, 331.87, 551.87, ..., 10055.1], calculate the energy ratio of the 1 / 2th fractional harmonic. 1_1 / 2 = [1, 0.0985, 0.00055, ..., 2.2721e-06]. The target energy ratio is v. 1_1 / 2 = [0.01617, 0.1294, 1, ..., 1.2708e-06]. Then the 1 / 2th order fractional harmonic energy adjustment ratio γ1_1 / 2 = [0.01617, 1.313, 1802.213, ..., 55.931], adjust the 1 / 2 harmonic energy ratio of y1(n) to obtain the spectrum Z. 1_1 / 2 (k). By successively setting β = 1 / 4 and 3 / 4, adjusting the fractional harmonic energy ratio of y1(n), the resulting spectrum is Z. 1_1 / 4 (k) and Z 1_3 / 4 (k). According to formula (18), the spectrum of each fractional harmonic is superimposed with the harmonic spectrum Z1(k) to obtain the spectrum Output1(k). Figure 7 It is the frequency domain logarithmic amplitude spectrum of y1(n) with 1 / 2, 1 / 4, and 3 / 4 fractional harmonics superimposed sequentially. After adding the fractional harmonics, it can be seen from the frequency domain logarithmic amplitude spectrum that the spectral energy at the fractional position increases.

[0153] The same fractional harmonics, including 1 / 2, 1 / 4, and 3 / 4 fractional harmonics, are added to the three frames of monotone audio in this embodiment to obtain the spectrum Output1(k), Output2(k), and Output3(k) of each frame.

[0154] S305.5: Output the frequency domain of the i-th frame. i (k) Fourier transform to the time domain to obtain the i-th frame time domain signal output after frequency domain feature transformation. i (n);

[0155] S306, K of the single-tone audio value y(n) y Steps S302 to S305 are repeated for each frame to obtain the time-domain signal output after frequency domain feature transformation of each frame. i (n), output i (n) is concatenated to obtain a single-tone audio output(n) with the timbre of the target instrument.

[0156] After frequency domain feature transformation, the spectrum of each frame is transformed to the time domain to obtain the time domain signals output1(n), output2(n), and output3(n) for each frame. For example... Figure 8 As shown, the time-domain waveform of the single-tone audio output(n) with the target timbre is obtained by splicing the time-domain signals of each frame.

[0157] S4. Repeat steps S2 and S3 for all single-note audio frequencies of the target instrument to complete the timbre conversion.

[0158] Example 2

[0159] This embodiment continues to provide the implementation process of the instrument timbre conversion method based on time-frequency characteristics, including the following steps:

[0160] S1. Referring to the corresponding steps in Example 1, extract timbre features from the instrument audio dataset and establish a target timbre feature library.

[0161] S2. Referring to the corresponding steps in Example 1, perform time-domain feature transformation on the audio to be converted to obtain the time-domain feature transformed monotone audio y(n);

[0162] S3. Referring to the corresponding steps in Example 1, perform frequency domain feature transformation on the single-tone audio y(n) to obtain the single-tone audio output(n) with the timbre of the target instrument.

[0163] S4. Referring to the corresponding steps in Example 1, repeat steps S2 and S3 for all single-tone audio frequencies of the target instrument to complete the timbre conversion.

[0164] This embodiment implements four types of instrument timbre conversion: piano to violin, violin to piano, piano to xylophone, and trumpet to flute. Different classifiers—Support Vector Machine (SVM), K-Nearest Neighbors (KNN), and ensemble learning—are used to perform binary classification of the converted audio to identify the instrument type. The recognition rates are shown in Table 2.

[0165] Table 2. Audio Recognition Rate Table

[0166] Source Musical Instruments Target instrument SVM KNN Ensemble Average recognition rate piano violin 60.0％ 60.0％ 80.1％ 66.7％ violin piano 79.1％ 84.2％ 73.7％ 78.8％ piano xylophone 87.5％ 87.5％ 90.7％ 88.6％ small flute 90.5％ 98.1％ 95.2％ 94.6％

[0167] Approximately 82.2% of the converted audio was correctly identified, indicating that the present invention can effectively convert the audio to be converted to the target instrument type.

[0168] Subjective instrument type identification was performed on the converted audio, and the recognition rate is shown in Table 3.

[0169] Table 3. Subjective Recognition Rate of Converted Audio

[0170]

[0171] The subjective recognition rate of the converted audio was 81.1%, indicating that the present invention can achieve a good timbre conversion effect.

[0172] The above embodiments are preferred embodiments of the present invention, but the embodiments of the present invention are not limited to the above embodiments. Any changes, modifications, substitutions, combinations, or simplifications made without departing from the spirit and principle of the present invention shall be considered equivalent substitutions and shall be included within the protection scope of the present invention.

Claims

1. A method for refining musical instrument timbre conversion based on time-frequency characteristics, characterized in that, The instrument timbre conversion method includes the following steps: S1. Extract timbre features from the instrument audio dataset and establish a target timbre feature library; wherein, the timbre features include fundamental frequency, fundamental pulse marker sequence, duration, temporal bilateral peak envelope, harmonic energy ratio, and fractional harmonic energy ratio; the fractional harmonic energy is defined as follows: when a certain frame of a single-tone audio... The frequency of the second harmonic is ,frequency Represented as The frequencies of fractional harmonics, among which , , For fractions, The value range is from 0 to 1. The harmonic order is given; the frequency is... The fractional harmonics are collectively referred to as Fractional harmonics, with a frequency of The energy ratio of fractional harmonics is collectively referred to as The proportion of fractional harmonic energy; S2. Based on the target instrument type, obtain the time-domain features of the target instrument's timbre from the target timbre feature library. Perform time-domain feature transformation on the fundamental frequency, duration, and time-domain amplitude envelope of the audio to be converted through fundamental frequency adjustment, duration modification, time-domain amplitude modulation, and fundamental frequency synchronization interpolation to obtain the time-domain feature-transformed single-tone audio. ; S3, for single-tone audio Frame segmentation and windowing are performed to extract the proportion of harmonic energy and the proportion of each fractional harmonic. Based on the target instrument type, the frequency domain characteristics of the target instrument's timbre are obtained from the target timbre feature library. The single-tone audio is then adjusted. The harmonic energy ratios and fractional harmonic energy ratios of each frame are subjected to frequency domain feature transformation; then, the audio frames are spliced ​​together to obtain a single-tone audio with the timbre of the target instrument. ; S4. Repeat steps S2 and S3 for all single-note audio frequencies of the target instrument to complete the timbre conversion.

2. The method for refining musical instrument timbre based on time-frequency characteristics according to claim 1, characterized in that, The musical instrument audio dataset refers to a dataset containing single-note audio of each musical instrument.

3. The method for refining musical instrument timbre based on time-frequency characteristics according to claim 1, characterized in that, The target timbre feature library is established by extracting the timbre features of each single-tone audio in the instrument audio dataset. Each single-tone audio of each instrument has a corresponding fundamental frequency, duration, fundamental pulse marker sequence, temporal bilateral peak envelope, harmonic energy ratio, and fractional harmonic energy ratio. The harmonic energy ratio and fractional harmonic energy ratio mentioned above refer to the harmonic energy ratio and fractional harmonic energy ratio of each frame obtained after dividing the single-tone audio into frames and adding windows.

4. The method for refining musical instrument timbre based on time-frequency characteristics according to claim 1, characterized in that, The process of step S1 is as follows: S101. Use a short-time energy-based dual-threshold endpoint detection method to remove the silent segments of single-tone audio from the dataset and obtain the duration feature, where the duration is the time length of the non-silent segment of the single-tone audio. S102. The fundamental frequency features of a single-tone audio are extracted using narrowband spectral energy frequency estimation based on time-domain autocorrelation and iterative resampling. First, a coarse estimate of the fundamental frequency of the instrument's single-tone audio is obtained through time-domain autocorrelation. Then, the coarse estimate is gradually approximated to the true fundamental frequency of the signal through iterative narrowband spectral energy frequency estimation, finally obtaining a fine estimate of the fundamental frequency of the single-tone audio. The specific steps are as follows: S102.1: First, the single-tone audio is divided into frames. One frame is then subjected to time-domain autocorrelation to obtain a coarse estimate of the fundamental frequency, denoted as . Assuming the audio signal has a sampling rate of ; S102.2: According to Determine the resampling rate First, determine the value of integer c according to formula (1); , formula (1), Compare and The size of the value, when , The calculation formula is as follows: , Official (2), when hour, , formula (3), S102.3: Use the determined resampling rate The frame is resampled using cubic spline interpolation; then the frame is... Windowing and performing discrete Fourier transform on the point yields its amplitude spectrum. ; S102.4: Amplitude spectrum Find the maximum peak value, and the peak value index is... ; S102.5: Based on The amplitude spectrum around a point before and after the point forms a narrow band. Referring to existing frequency estimation formulas based on narrow band spectral energy, the frequency of the single-tone audio frequency is estimated. : , formula (4); S102.6: According to Sure Harmonic order values: , formula (5), in () is the rounding function. express For the first Second harmonics; S102.7: Obtain the new baseband The fundamental frequency at this time is: , formula (6), S102.8: Iteratively estimate the fundamental frequency; set a threshold. When the difference between the fundamental frequencies of two consecutive iterations is greater than a threshold ,Right now ,by Repeat step S102.2; when the difference between the fundamental frequencies of two adjacent iterations is less than the threshold... ,Right now , Then the fundamental frequency of the audio signal is infinitely approximated. This is the precise estimate of the fundamental frequency of a single-tone audio frequency, denoted as . Threshold The smaller the value, the closer the estimated frequency is to the true fundamental frequency of the audio signal; S103 performs pitch pulse tagging on single-tone audio to obtain the pitch pulse tagging sequence. The specific steps are as follows: S103.1: The period of a single tone is denoted as... On the time-domain waveform of a single-tone audio signal, determine three consecutive periods of the non-silent segment, i.e. ( ), ( )and( The location of the maximum peak point is denoted as . , and ,in Indicates the first The position of the marker point in each cycle; first let Perform the following steps; S103.2: According to , , Determine the number of sampling points for the fundamental period , Difference in the number of sampling points between two adjacent periods ; Official (7), , Official (8), , Official (9), S103.3: Predict the position of the next synchronization marker: , formula (10), Find the local maxima near this location, and mark the locations corresponding to the local maxima. ; Calculate the first Number of sampling points per cycle : , formula (11), Continue with steps S103.2 and S103.3 to find the next marker point. Marking ends when the predicted marker point position exceeds the duration of a single tone. Obtain the pitch pulse marker sequence, denoted as... ,in The number of cycles; S104. A peak detection method based on a sliding window is used to obtain the temporal bilateral peak envelope features of a single-tone audio. The process is as follows: Define a window with a length equal to the temporal period length of the single-tone audio. The moving length is 2 / 3, find the maximum and minimum points of each window, and then obtain the temporal bilateral peak envelope of the monotone audio through spline fitting; S105. Resample the single-tone audio, divide it into frames, and determine the harmonic frequency sequence of each frame. The specific process is as follows: Window each frame of single-tone audio, obtain the amplitude spectrum through Fourier transform, and determine the harmonic frequency sequence of each frame based on the fundamental frequency of the single-tone audio. At a frequency of Find the frequency corresponding to the peak point of the amplitude spectrum nearby, and use it as the first frequency. The frequency of subharmonics ;make , to obtain The frequency sequence of the second harmonic is denoted as ; S106. Based on the harmonic frequency sequence of each frame, calculate the proportion of harmonic energy for each frame. The specific process is as follows: First, based on the harmonic frequency sequence of each frame, calculate the proportion of harmonic energy for each frame. As the first The center frequency of the second harmonic is taken as the bandwidth, which is the fundamental frequency. 10% of Spectral energy in the frequency range as harmonics The energy of each harmonic is calculated to obtain the harmonic energy ratio of each frame. S107. Calculate the harmonic frequency sequence for each frame. fractional harmonic frequency sequence, to obtain The proportion of fractional harmonic energy is determined as follows: Each frequency of the fractional harmonic frequency sequence The bandwidth is taken as the fundamental frequency, which is used as the center frequency. 10% of Spectral energy in the frequency range as The energy, of which express The fractional harmonic frequency sequence Each frequency, calculated The harmonic energy of each frequency of the fractional harmonics is used to obtain the value of each frame. The proportion of fractional harmonic energy. The values ​​are 1 / 2, 1 / 4, 3 / 4, 1 / 8, and 7 / 8. S108. Obtain the timbre features of each single-note audio in the instrument audio dataset through the above steps, and establish a target timbre feature library.

5. The musical instrument timbre conversion method based on refined time-frequency features according to claim 4, characterized in that, The process of step S2 is as follows: S201. Adjust the fundamental frequency of the audio to be converted. The process is as follows: Repeat step S102 to obtain the fundamental frequency of the audio to be converted. Based on the fundamental frequency characteristics of the target musical instrument By interpolating and adjusting the fundamental frequency of the audio to be converted, a single-tone audio with adjusted fundamental frequency can be obtained. , It is an audio with transitional timbre, in which cubic spline interpolation is used; S202, Adjust by deleting or inserting pitch pulse marker units The duration of the note is determined by the following process: Repeat the duration acquisition method in step S101 to obtain the duration. The duration of the sound is denoted as Then, based on the target base frequency right Pitch pulse marking is performed, and windowing is applied to each marking period to obtain a short-time frame sequence of the pitch; based on the single-tone audio... The duration of the note and the duration of the target instrument Calculate the pitch length modification ratio, delete and insert new pitch short frames, and complete the processing of single-tone audio. Modify the duration of the note to obtain the single-note audio. , It is an audio with transitional timbre; S203, According to the target base frequency For single-tone audio Pitch pulse marking is performed, and amplitude modulation is applied to each marking cycle to achieve a single-tone audio. The temporal envelope adjustment process is as follows: First, the upper and lower envelope peak values ​​of each marked period are calculated. Then, the scaling factor is calculated based on the temporal peak envelope characteristics of the target instrument to achieve upper and lower envelope amplitude modulation. Next, the waveforms of each period after amplitude modulation are spliced ​​together. Finally, a Gaussian filter is used for smoothing to obtain the single-tone audio after duration adjustment. , It is an audio with transitional timbre; S204. Based on the target pitch pulse marker sequence, analyze the single-tone audio... The process of performing pitch synchronization interpolation is as follows: based on the target fundamental frequency... For single-tone audio A pitch pulse labeling sequence is generated, and then the number of sampling points for each pitch period of the target instrument is obtained based on the characteristics of the pitch pulse labeling sequence of the target instrument; interpolation is then used to make... The number of sampling points in each cycle is consistent with the number of sampling points in each cycle of the target instrument, which is pitch-synchronized interpolation to obtain single-tone audio. , The interpolation method used is cubic spline interpolation.

6. The musical instrument timbre conversion method based on refined time-frequency features according to claim 1, characterized in that, The process of step S3 is as follows: S301, for single-tone audio Resampling and frame-by-frame windowing are performed to obtain the harmonic frequency sequence of each frame of the signal. The resampling method uses cubic spline interpolation, and the process is as follows: based on the target fundamental frequency... Determine the resampling rate for single-tone audio. Perform resampling, framing, and windowing; convert single-tone audio... No. The time-domain signal of a frame is denoted as , , For frame count, get The harmonic frequency sequence; S302, Calculation Obtain the proportion of each harmonic energy. The proportion of harmonic energy, denoted as ; S303, Extraction The time-domain waveforms of each harmonic are obtained using the following process: A window-function-based FIR bandpass filter is used to obtain... The time-domain signals of each harmonic will The The time-domain signal of each harmonic is denoted as ; S304, According to the target instrument Harmonic energy ratio of frame signal ,Adjustment The proportions of each harmonic energy, and the process are as follows: Will The spectrum is obtained by performing a Fourier transform. According to the target instrument Harmonic energy ratio of frames Acquired from S302 Harmonic energy ratio ;Will The The coefficients are denoted as ,Will The The coefficients are denoted as The harmonic energy ratio adjustment coefficient is calculated according to formula (12). : Official (12), in, Adjusting the energy ratio The Each harmonic energy adjustment coefficient is used to adjust the energy of each harmonic according to formula (13): Official (13), in, For the first energy adjustment Frame number The spectrum of the 1st harmonic; according to formula (14), the 1st harmonic... The harmonic spectrum is obtained by superimposing the spectra of each harmonic in the frame. , ; , formula (14), S305, According to the target instrument Frame signal fractional harmonic energy ratio ,Adjustment The proportion of fractional harmonic energy is determined as follows: S305.1: According to Determination of harmonic frequency sequence of fractional harmonic frequency sequence, to obtain fractional harmonic energy ratio and The time-domain signals of the fractional harmonics are processed as follows: The fractional harmonic energy ratio acquisition section is used to obtain... of fractional harmonic energy ratio ,exist The time-domain signal components of the fractional harmonics are filtered out using an FIR bandpass filter. middle The time-domain signals of each frequency of the fractional harmonics will fractional harmonics The time-domain signal is denoted as ; S305.2: According to the target instrument's... Frames fractional harmonic energy ratio ,Adjustment middle The proportion of fractional harmonic energy is determined as follows: Will The The coefficients are denoted as ,Will The The coefficients are denoted as Calculate according to formula (15) Fractional harmonic energy adjustment ratio coefficient : , formula (15), Will Perform a Fourier transform to obtain the spectrum. Adjust according to formula (16) Energy of fractional harmonics: Official (16), in, Adjusting the energy ratio The One energy adjustment coefficient; For the first energy adjustment frame fractional harmonics One spectrum; According to formula (17) The spectra of the fractional harmonics are superimposed to obtain a result containing only the following frequencies. Spectrum of fractional harmonics ; , Official (17); S305.3: Make For each fraction greater than 0 and less than 1, repeat steps S305.1 to S305.2 to obtain the spectrum of each fractional harmonic; S305.4: According to formula (18), the spectrum of each fractional harmonic is... With harmonic spectrum Superimpose; Official (18), S305.5: The first Frame frequency domain Fourier transform to the time domain yields the frequency domain feature transformation result. Frame time domain signal ; S306, Single-tone audio of Steps S302 to S305 are repeated for each frame to obtain the time-domain signal after frequency domain feature transformation for each frame. ,Will By splicing the audio, a single-note audio with the timbre of the target instrument can be obtained. .

Images