Audio processing method, device and readable storage medium

Abstract

The present application discloses an audio processing method, device and readable storage medium, the method comprising: obtaining initial audio, extracting the initial spectrum of the initial audio and the spectrum envelope corresponding to the timbre; performing scaling processing on the spectrum envelope to obtain a target spectrum envelope; using the target spectrum envelope and the spectrum envelope to obtain a weight coefficient; using the weight coefficient, processing the initial spectrum to obtain a target spectrum; performing inverse Fourier transform on the target spectrum to obtain a target audio after the timbre is changed. In the present application, since scaling processing is performed on the spectrum envelope corresponding to the timbre will not cause synchronous changes in pitch, the target audio obtained after the timbre change processing will not have the problem of synchronous changes in pitch. That is, the difference between the target audio after the timbre change and the initial audio is only a change in timbre, but no change in pitch, so the target audio will be more realistic.

Description

Technical Field

[0001] The present application relates to the field of sound processing technology, and in particular to an audio processing method, device and readable storage medium. Background Art

[0002] Sound has three characteristics: pitch, loudness and timbre. Among them, timbre is determined by factors such as the material and structure of the sound-generating body. Changing the timbre of sound can bring users a new auditory experience and increase the fun of sound-related applications / devices.

[0003] By performing pitch-shifting operations on sounds, different timbres can be obtained. However, when using pitch-shifting operations to change the timbre, the pitch will inevitably be modified synchronously, which results in not only a change in timbre but also a change in pitch after the change, which does not meet user expectations. For example, when a person's voice is changed in timbre through pitch-shifting operations, the pitch will also change synchronously, which will cause the sound after the timbre change to be distorted, that is, the sound after the timbre change does not sound like the person's voice at all, or even like a human voice.

[0004] In summary, how to effectively change the timbre of audio and other issues are technical problems that technical personnel in this field urgently need to solve. Summary of the invention

[0005] The purpose of this application is to provide an audio processing method, device and readable storage medium, which can change the timbre without changing the pitch by advancing the spectral envelope corresponding to the timbre in the audio and then only scaling the spectral envelope corresponding to the timbre.

[0006] In order to solve the above technical problems, this application provides the following technical solutions:

[0007] In a first aspect, the present application provides an audio processing method, comprising:

[0008] Acquire initial audio, and extract an initial spectrum of the initial audio and a spectrum envelope corresponding to the timbre;

[0009] Performing scaling processing on the spectrum envelope to obtain a target spectrum envelope;

[0010] Obtaining a weight coefficient using the target spectrum envelope and the spectrum envelope;

[0011] Using the weight coefficients, processing the initial spectrum to obtain a target spectrum;

[0012] Perform inverse Fourier transform on the target frequency spectrum to obtain the target audio with changed timbre.

[0013] In an optional specific implementation manner, performing scaling processing on the spectrum envelope to obtain a target spectrum envelope includes:

[0014] Get age simulation parameters from the client;

[0015] According to the negative correlation between the age simulation parameter and the stretch-shrink coefficient, determining the stretch-shrink coefficient corresponding to the age simulation parameter;

[0016] The spectrum envelope is scaled according to the scaling factor to obtain the target spectrum envelope.

[0017] In an optional specific implementation manner, the obtaining of age simulation parameters from the client includes:

[0018] Obtaining a reference age and a timbre simulation age corresponding to the initial audio;

[0019] The age simulation parameter is determined based on a ratio of the reference age to the timbre simulation age.

[0020] In an optional specific implementation manner, performing scaling processing on the spectrum envelope according to the scaling factor to obtain the target spectrum envelope includes:

[0021] If the stretch coefficient is less than 1, stretching the spectrum envelope to obtain the target spectrum envelope with brighter timbre;

[0022] If the stretch coefficient is greater than 1, the spectrum envelope is compressed to obtain the target spectrum envelope with a darker timbre.

[0023] In an optional specific implementation manner, using the target spectrum envelope and the spectrum envelope to obtain a weight coefficient includes:

[0024] Differentiating the target spectrum envelope and the spectrum envelope to obtain initial mask spectrum coefficients;

[0025] The initial Mask spectrum coefficients are mapped to a linear domain, and mask smoothing is performed using a triangular window convolution to obtain the weight coefficients.

[0026] In an optional specific implementation manner, the target spectrum envelope and the spectrum envelope are differentiated to obtain initial mask spectrum coefficients, including:

[0027] If the valid data interval of the target spectrum envelope is within the data interval of the spectrum envelope, the target spectrum envelope and the spectrum envelope are differentiated, and the differential value is determined as the initial mask spectrum coefficient;

[0028] If the valid data interval of the target spectrum envelope exceeds the data interval of the spectrum envelope, the corresponding initial mask spectrum coefficient is determined to be zero.

[0029] In an optional specific implementation manner, the initial spectrum is processed using the weight coefficient to obtain a target spectrum, including:

[0030] The weight coefficient is multiplied by the initial spectrum to obtain the target spectrum.

[0031] In an optional specific implementation, performing inverse Fourier transform on the target spectrum to obtain the target audio with changed timbre includes:

[0032] Supplementing the target spectrum with corresponding Hening window weight coefficients;

[0033] Performing an inverse short-time Fourier transform on the target spectrum supplemented with the Henin window weight coefficients to obtain modulation of each frame;

[0034] The pitch changes of the frames are overlapped and added to obtain the target audio.

[0035] In a second aspect, the present application provides an audio processing device, comprising:

[0036] A spectrum and spectrum envelope extraction module, used to obtain initial audio, extract an initial spectrum of the initial audio and a spectrum envelope corresponding to the timbre;

[0037] An envelope processing module, used for performing expansion and contraction processing on the spectrum envelope to obtain a target spectrum envelope;

[0038] A weight coefficient determination module, used to obtain a weight coefficient using the target spectrum envelope and the spectrum envelope;

[0039] A spectrum processing module, used to process the initial spectrum using the weight coefficients to obtain a target spectrum;

[0040] The spectrum conversion module is used to perform inverse Fourier transform on the target spectrum to obtain the target audio with changed timbre.

[0041] In a third aspect, the present application provides an electronic device, including:

[0042] Memory for storing computer programs;

[0043] A processor is used to implement the steps of the above-mentioned audio processing method when executing the computer program.

[0044] In a fourth aspect, the present application provides a readable storage medium having a computer program stored thereon, and the computer program implements the steps of the above-mentioned audio processing method when executed by a processor.

[0045] Apply the method provided in the embodiment of the present application to obtain initial audio, extract the initial spectrum of the initial audio and the spectrum envelope corresponding to the timbre; perform scaling processing on the spectrum envelope to obtain a target spectrum envelope; use the target spectrum envelope and the spectrum envelope to obtain a weight coefficient; use the weight coefficient to process the initial spectrum to obtain a target spectrum; perform inverse Fourier transform on the target spectrum to obtain the target audio after the timbre is changed.

[0046] The initial audio is obtained, the initial spectrum of the initial audio and the spectrum envelope corresponding to the timbre are extracted, and the target spectrum envelope corresponding to the timbre change can be obtained by performing a stretching process on the spectrum envelope. Then, based on the spectrum envelope and the target spectrum envelope, a weight coefficient for processing the initial spectrum and realizing the timbre change is determined. The weight coefficient is applied to the initial spectrum to obtain the target spectrum. Finally, the target audio after the timbre change can be obtained by performing an inverse Fourier transform on the target spectrum. It can be seen that in the present application, the timbre change is not achieved by pitch change, but the spectrum envelope corresponding to the timbre is directly stretched. Since the spectrum envelope corresponding to the timbre is stretched and retracted, the pitch will not change synchronously, and thus the target audio obtained after the timbre change process will not have the problem of synchronous change in pitch. That is, the difference between the target audio after the timbre change and the initial audio is only the timbre change, but no pitch change, so the target audio will be more realistic.

[0047] Correspondingly, the embodiments of the present application also provide an audio processing device, equipment and readable storage medium corresponding to the above-mentioned audio processing method, which have the above-mentioned technical effects and are not repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the related technologies, the drawings required for use in the embodiments or the related technical descriptions are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0049] Figure 1 This is a flowchart of an implementation of an audio processing method in an embodiment of the present application;

[0050] Figure 2 This is a schematic diagram of the structure of an audio processing device in an embodiment of the present application;

[0051] Figure 3 This is a schematic diagram of the structure of an electronic device in an embodiment of the present application;

[0052] Figure 4This is a schematic diagram of the specific structure of an electronic device in an embodiment of the present application. DETAILED DESCRIPTION

[0053] In order to enable those skilled in the art to better understand the present application, the present application is further described in detail below in conjunction with the accompanying drawings and specific implementation methods. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of the present application.

[0054] The audio processing method provided in this application can be applied to electronic devices that need to process the timbre of sound, including but not limited to servers, PCs and mobile terminals. The specific internal structure of the electronic device can be referred to Figure 3 and Figure 4 And the corresponding descriptions are not repeated here one by one. Specifically, since the audio processing method provided by this application can change the timbre without changing the pitch, it can further display or restore the timbre differences of the currently input human voice at different ages through controlled changes in the timbre, and can be applied to products such as Kugou. For example, in the application of virtual character display, when the age of the character changes, the timbre is adaptively changed to correspond to the vocal output of different age stages, so as to be closer to the display of real people.

[0055] Please refer to Figure 1 , Figure 1 This is a flowchart of an audio processing method in an embodiment of the present application, the method comprising the following steps:

[0056] S101, obtaining initial audio, extracting an initial spectrum of the initial audio and a spectrum envelope corresponding to the timbre.

[0057] The initial audio may specifically correspond to a human voice (including a real human voice or a simulated human voice), or may correspond to other sounds, such as musical instrument sounds (such as trumpet sounds, etc.), or natural sounds (such as insect chirping and bird calls, etc.). In this embodiment, the initial audio is the audio that needs to be processed for timbre change, and there is no specific limitation on what sound the initial audio specifically corresponds to.

[0058] The initial audio may be directly collected by a sound collection device, or may be directly read from a storage medium, or may be received from other devices. Specifically, the initial audio may be obtained by receiving a request. The timbre change request may be specifically initiated by some terminals or applications that need to change the timbre. The timbre change request may specifically carry the initial audio to be changed in timbre or the marking information of the initial audio. That is to say, after receiving the timbre change request, the timbre change request may be parsed to obtain the initial audio to be changed in timbre.

[0059] After obtaining the initial audio, the initial spectrum of the initial audio and the spectrum envelope corresponding to the timbre can be extracted.

[0060] Sound includes three characteristics: pitch, timbre and loudness. In this embodiment, the timbre characteristic is focused on. Therefore, in this embodiment, after obtaining the initial audio, the initial spectrum of the initial audio and the corresponding characteristics of the timbre are extracted. Specifically, the characteristics corresponding to the timbre can be characterized by the spectrum envelope related to the timbre. Therefore, in this embodiment, the spectrum envelope related to the timbre in the initial audio needs to be extracted.

[0061] To extract the initial audio, Fourier transform can be used to process the initial audio, converting it from the time domain to the frequency domain to obtain the initial spectrum.

[0062] How to extract the spectrum envelope corresponding to the initial audio and timbre may be specifically referred to how to extract the corresponding spectrum envelope from a signal.

[0063] In order to better extract the spectrum envelope corresponding to the timbre, in a specific implementation of the present application, the spectrum envelope corresponding to the timbre in the initial audio may be extracted by referring to the following steps:

[0064] Step 1: extract the baseband synchronization sequence of the initial audio, divide the baseband synchronization sequence into frames, and add a Haining window to obtain a frame signal sequence after segmentation;

[0065] Step 2: Perform short-time Fourier transform on the frame signal sequence to obtain the spectrum distribution of the initial audio;

[0066] Step 3: Using the frequency spectrum distribution, obtain the power spectrum curve;

[0067] Step 4: Extract the signal envelope of the power spectrum curve, and use the signal envelope to determine the spectrum envelope.

[0068] For ease of description, the above four steps are combined for explanation below.

[0069] Among them, the Hanning window, also known as the Hanning window, is one of the window functions and a special case of the raised cosine window. The Hanning window can be regarded as the sum of the spectra of three rectangular time windows, or the sum of three sinc(t) type functions, and the two items in the brackets are moved to the left and right by π / T relative to the first spectrum window, so that the side lobes cancel each other and eliminate high-frequency interference and leakage energy. It is suitable for non-periodic continuous signals.

[0070] For ease of processing, in this embodiment, the baseband synchronization sequence of the initial audio is first extracted. Specifically, in practical applications, a baseband extraction tool such as the PYin open source tool, crepe, and harvest can be used to extract the baseband synchronization sequence. Among them, the PYin open source tool, crepe, and harvest are all existing baseband extraction tools.

[0071] The following uses the PYin open source tool as an example to explain in detail how to extract the baseband synchronization sequence, process the extracted baseband synchronization sequence, and finally obtain the spectrum envelope corresponding to the timbre.

[0072] Assume that the initial audio is: x(i), i=0, 1, 2, ... represents the sample point index, use the PYin open source tool to first extract the fundamental frequency synchronization sequence, that is, the fundamental frequency synchronization information, referred to as F0 sequence (where f0 corresponds to Pitch(f0), that is, fundamental frequency, fundamental tone) from the initial audio at intervals of 5ms (of course, other time lengths can also be used as intervals, such as 4ms, 6ms, etc.).

[0073] Based on the signal's baseband synchronization information (i.e., F0 sequence), the baseband period sequence is 3 times (3T0, of course, in practical applications, other multiples can also be used as the window length, which will not be listed here) as the window length (1.5T0 on each side), the signal is framed, and a Hanning window is added to obtain the segmented frame signal sequence. The spectrum distribution is obtained through STFT (short-time Fourier transform): Where n represents the frame index, ω represents the digital angular frequency variable of the frame, F represents the short-time Fourier transform, and W hannn (i) indicates the Haining window.

[0074] in, represents Fourier transform, L represents the offset of the inter-frame signal, and the number of samples corresponding to 5ms can be selected (it should be noted that other values ​​can also be selected as the number of samples, which will not be listed here). The Hanning window is defined as:

[0075] Here N=3T0.

[0076] Calculate the power spectrum curve:

[0077] P(ω,n)=||X(ω,n)||2 .

[0078] Calculate the power spectrum of each frame signal and use a rectangular window to linearly smooth the power spectrum, that is:

[0079] Among them, the fundamental frequency characteristic length of the rectangular window is:

[0080] Then, based on this, the timbre (i.e., the resonance peak envelope information) is extracted: the power spectrum is regarded as an ordinary signal, and the signal envelope is obtained. That is, the low-frequency part is found. The weight of the cepstrum segment close to the gene period (fundamental pitch period) in the cepstrum domain is increased to obtain a more realistic spectrum envelope curve. The calculation process is described as follows (for the convenience of description, the frame index parameter n is omitted in the following):

[0081] Smoothed logarithmic power spectrum:

[0082]

[0083] Cepstrum:

[0084]

[0085] Construct a cepstrum lifting window:

[0086]

[0087] Where sinc(·) represents the sampling function, and q can take an empirical value, such as q = -0.09; Cepstrum windowing: Logarithmic spectrum after spectrum enhancement: in: They represent forward and reverse Fourier transform respectively.

[0088] After the above processing, the logarithmic power spectrum is finally obtained. It can be regarded as the logarithmic resonance peak envelope of the signal. Since the envelope information currently obtained actually represents the logarithmic power spectrum, on this basis, the amplitude logarithmic spectrum can be directly obtained by weighting by 0.5 times, that is, the spectrum envelope corresponding to the timbre is expressed as:

[0089]

[0090] S102: Perform scaling processing on the spectrum envelope to obtain a target spectrum envelope.

[0091] After obtaining the spectrum envelope corresponding to the timbre, the spectrum envelope can be stretched and retracted to obtain the target spectrum envelope.

[0092] In this embodiment, there is no limitation on the specific extension or compression of the spectrum envelope, the degree of extension, or the degree of compression. The target spectrum envelope obtained after the spectrum envelope is stretched is different from the spectrum envelope, that is, the spectrum envelope and the target spectrum envelope correspond to spectrum envelopes of different timbres.

[0093] S103: Obtain weight coefficients using the target spectrum envelope and the spectrum envelope.

[0094] After the target spectrum envelope is obtained, since the target spectrum envelope and the spectrum envelope correspond to different timbres, based on the target spectrum envelope and the spectrum envelope, a weight coefficient for processing the initial spectrum to match the spectrum envelope after the timbre change can be obtained. The weight coefficient is the spectrum that can process the initial spectrum to correspond to the target spectrum envelope.

[0095] S104: Using the weight coefficients, the initial spectrum is processed to obtain a target spectrum.

[0096] After the weight coefficients are obtained, the weight coefficients can be applied to the initial spectrum to obtain the target spectrum, which is the spectrum that matches the envelope of the target spectrum.

[0097] Specifically, the weight coefficient may be multiplied by the initial spectrum to obtain the target spectrum.

[0098] S105, performing inverse Fourier transform on the target frequency spectrum to obtain the target audio with changed timbre.

[0099] After obtaining the target spectrum, the target spectrum can be converted from the frequency domain to the time domain through inverse Fourier transform, thereby obtaining the target audio with changed timbre.

[0100] In this embodiment, for the sake of easy distinction, the initial audio after the timbre change is referred to as the target audio, that is, the difference between the target audio and the initial audio is only that the corresponding timbre is different.

[0101] Apply the method provided in the embodiment of the present application to obtain initial audio, extract the initial spectrum of the initial audio and the spectrum envelope corresponding to the timbre; perform scaling processing on the spectrum envelope to obtain a target spectrum envelope; use the target spectrum envelope and the spectrum envelope to obtain a weight coefficient; use the weight coefficient to process the initial spectrum to obtain a target spectrum; perform inverse Fourier transform on the target spectrum to obtain the target audio after the timbre is changed.

[0102] The initial audio is obtained, the initial spectrum of the initial audio and the spectrum envelope corresponding to the timbre are extracted, and the target spectrum envelope corresponding to the timbre change can be obtained by performing a stretching process on the spectrum envelope. Then, based on the spectrum envelope and the target spectrum envelope, a weight coefficient for processing the initial spectrum and realizing the timbre change is determined. The weight coefficient is applied to the initial spectrum to obtain the target spectrum. Finally, the target audio after the timbre change can be obtained by performing an inverse Fourier transform on the target spectrum. It can be seen that in the present application, the timbre change is not achieved by pitch change, but the spectrum envelope corresponding to the timbre is directly stretched. Since the spectrum envelope corresponding to the timbre is stretched and retracted, the pitch will not change synchronously, and thus the target audio obtained after the timbre change process will not have the problem of synchronous change in pitch. That is, the difference between the target audio after the timbre change and the initial audio is only the timbre change, but no pitch change, so the target audio will be more realistic.

[0103] It should be noted that, based on the above embodiments, the embodiments of the present application also provide corresponding improved solutions. In the preferred / improved embodiments, the same steps or corresponding steps as those in the above embodiments can be referenced to each other, and the corresponding beneficial effects can also be referenced to each other, which will not be described one by one in the preferred / improved embodiments of this article.

[0104] In a specific implementation of the present application, when the spectrum envelope is stretched, the spectrum envelope can be stretched in a directional and targeted manner, so that the change in timbre is also directional and targeted. Specifically, the above step S102 stretches the spectrum envelope to obtain the target spectrum envelope, which specifically includes:

[0105] Step 1: Obtain age simulation parameters from the client;

[0106] Step 2: According to the negative correlation between the age simulation parameter and the stretch-shrink coefficient, determine the stretch-shrink coefficient corresponding to the age simulation parameter.

[0107] Step 3: Perform scaling processing on the spectrum envelope according to the scaling factor to obtain the target spectrum envelope. For ease of description, the above three steps are combined for explanation below.

[0108] The spectrum envelope can be processed by STFT (Short Time Fourier Transform). Here, a fixed frame length of 2048 points is used (corresponding to 46.4ms when the sampling rate is 44100), and the frame shift is 5ms (the same as the previous envelope extraction frame shift). At this time, the hanning window length N=2048, and then the stretch coefficient corresponding to the amplitude log spectrum is calculated.

[0109] Research has found that the timbre of the same person generally changes at different ages, and the timbre changes show a certain pattern. From the auditory point of view, the human voice is relatively light in youth, and the overall timbre characteristics tend to be in the middle and high frequency direction. With the increase of age, the voice moves closer to the low frequency band. From the perspective of the timbre corresponding to the spectrum envelope, the younger the person, the larger the lateral amplitude of the spectrum envelope of the corresponding timbre; and the older the person, the smaller the lateral amplitude of the spectrum envelope of the corresponding timbre. Based on this, in this application, according to this characteristic, the final timbre performance of the human voice is modified by adjusting the spectrum distribution of the resonance peak envelope.

[0110] Specifically, the age simulation parameters can be obtained from the client, and then the stretch coefficients can be determined based on the negative correlation between the age simulation parameters and the stretch coefficients. Then, the spectrum envelope is scaled according to the stretch coefficients to obtain the target spectrum envelope that matches the age simulation parameters.

[0111] Among them, the larger the age simulation parameter is, the smaller the expansion and contraction coefficient is.

[0112] That is to say, in this embodiment, the spectrum envelope is scaled by using the timbre brightness characteristics of human voice at different ages. Specifically, since human voice is relatively light in youth, the overall timbre characteristics tend to be in the middle and high frequency direction. As age increases, the voice is closer to the low frequency band. Based on this characteristic, this embodiment modifies the final timbre performance of human voice by adjusting the spectrum distribution of the formant envelope.

[0113] It should be noted that in this application, after research, it was found that if fshift is used to characterize the stretch coefficient, in order to avoid an exaggerated change in timbre, the stretch coefficient is generally in the range of fshift∈[0.8655,1.1554], and when fshift>1, the curve is expanded; when fshift<1, the curve is compressed. Therefore, the coefficient interval corresponding to the compression process can be specifically [0.8655,1), and the coefficient interval corresponding to the stretch process can be specifically (1,1.1554]. In other words, when timbre aging processing is to be performed, the corresponding stretch coefficient can be determined from [0.8655,1). Specifically, the stretch coefficient can be determined by random selection, or by setting different values ​​corresponding to different age groups, or by percentage from this interval. Of course, the ratio of the reference age to the timbre simulation age can also be used for confirmation. Correspondingly, for rejuvenation, the determination method of aging can also be referred to, which will not be elaborated here.

[0114] Wherein, obtaining age simulation parameters from the client may specifically include:

[0115] Step 1, obtaining the reference age and timbre simulation age corresponding to the initial audio;

[0116] Step 2: Determine age simulation parameters based on the ratio of reference age to timbre simulation age.

[0117] The reference age can be specifically the reference age determined by identifying the timbre feature after extracting the timbre feature of the initial audio. Of course, the reference age can also be the real age entered by the user. For aging, the frequency envelope needs to be compressed, and for rejuvenation, the spectrum envelope needs to be widened. Therefore, the age simulation parameter can be determined based on the ratio of the reference age to the timbre simulation age. In other words, when the reference age is greater than the timbre simulation age, it is rejuvenation, and the age simulation parameter is greater than 1, while the stretch coefficient is less than 1; when the reference age is less than the timbre simulation age, it is aging, and the age simulation parameter is less than 1, while the stretch coefficient is greater than 1. When determining the stretch coefficient, the stretch coefficient can be determined by referring to the negative correlation between the simulation parameter and the stretch coefficient. That is, the larger the age simulation parameter, the smaller the stretch coefficient. In this application, when the age simulation parameter is negative, the stretch coefficient is positive, and vice versa. The corresponding degree of negative correlation can be set according to actual needs.

[0118] Among them, step three is to perform scaling processing on the spectrum envelope according to the scaling factor to obtain the target spectrum envelope. It can be divided into the following two cases:

[0119] Case 1: If the stretch coefficient is less than 1, the spectrum envelope is stretched to obtain a target spectrum envelope with brighter timbre;

[0120] Case 2: If the stretch coefficient is greater than 1, the spectrum envelope is compressed to obtain a target spectrum envelope with a darker timbre.

[0121] The frequency spectrum envelope is stretched and contracted according to the stretch coefficient to obtain the target frequency spectrum envelope, which may be specifically: the frequency spectrum envelope is interpolated and stretched according to the stretch coefficient to obtain the target frequency spectrum envelope.

[0122] The expansion and contraction of the curve is achieved here by interpolation: Ly(ω)=interp(Lx(ω),fshift), where the symbol interp(·) represents the interpolation operation of the curve, and open source tools can be used for linear interpolation or polynomial interpolation. fshift represents the expansion and contraction coefficient.

[0123] In a specific implementation of the present application, the above step S103 uses the target spectrum envelope and the spectrum envelope to obtain the weight coefficient, including:

[0124] Step 1: Differentiate the target spectrum envelope and the spectrum envelope to obtain the initial mask spectrum coefficients;

[0125] Step 2: Map the initial Mask spectrum coefficients to the linear domain, and use triangular window convolution to perform mask smoothing to obtain weight coefficients.

[0126] For ease of description, the above two steps are combined for explanation below.

[0127] First, the logarithmic spectrum difference is used, that is, the spectrum envelope and the target spectrum envelope are differentiated to obtain the initial Mask spectrum coefficient (that is, the initial mask spectrum coefficient): LM(ω)=Ly(ω)-Lx(ω);

[0128] Then it is mapped to the linear domain: Mask(ω)=exp(LM(ω)).

[0129] Specifically, the above step 1 specifically includes the following two situations:

[0130] Case 1: If the valid data interval of the target spectrum envelope is within the data interval of the spectrum envelope, the target spectrum envelope and the spectrum envelope are differentiated, and the differential value is determined as the initial mask spectrum coefficient;

[0131] Case 2: if the valid data interval of the target spectrum envelope exceeds the data interval of the spectrum envelope, the corresponding initial mask spectrum coefficient is determined to be zero.

[0132] That is to say, when fshift>1, the valid data interval of the curve Ly(ω) after the curve expansion exceeds the data interval of the original curve Lx(ω), and Mask(ω)=0 is defined at this time.

[0133] In order to prevent the weight coefficient from jittering too much, the triangular window convolution can be used to achieve Mask smoothing: M(ω) = conv(Mask(ω),W tri (ω)), where Mask(ω)=conv(·) represents the convolution process, W tri (ω) represents the triangular window function. The current window function is composed of three points: W tri (ω)=[0.25,0.50,0.25].

[0134] In a specific implementation of the present application, step S105 performs inverse Fourier transform on the target spectrum to obtain the target audio with changed timbre, which specifically includes:

[0135] Step 1: Supplement the corresponding Hening window weight coefficient to the target spectrum;

[0136] Step 2: Perform short-time Fourier inverse transform on the target spectrum supplemented with the Henin window weight coefficient to obtain the modulation of each frame;

[0137] Step 3: Overlap and add the pitch changes of each frame to obtain the target audio.

[0138] For ease of description, the above three steps are combined for explanation below.

[0139] After applying the weight coefficient M(ω) to the initial spectrum to obtain the weighted target spectrum of the current frame: Y(ω,n)=M(ω,n)·X(ω,n), the inverse short-time Fourier transform can be used to supplement the corresponding hann window weight coefficient to obtain the modulated output of the current frame: Where i represents the time domain (sample point) sequence, and n represents the frame sequence.

[0140] Finally, the overlap-add (OLA) method is used to obtain the final target audio: y(i)=OLA{y(i,n)}.

[0141] Corresponding to the above method embodiment, the embodiment of the present application further provides an audio processing device. The audio processing device described below and the audio processing method described above can refer to each other.

[0142] See also Figure 2 As shown, the device includes the following modules:

[0143] The spectrum and spectrum envelope extraction module 101 is used to obtain the initial audio, extract the initial spectrum of the initial audio and the spectrum envelope corresponding to the timbre;

[0144] The envelope processing module 102 is used to perform scaling processing on the spectrum envelope to obtain a target spectrum envelope;

[0145] The weight coefficient determination module 103 is used to obtain the weight coefficient using the target spectrum envelope and the spectrum envelope;

[0146] The spectrum processing module 104 is used to process the initial spectrum using the weight coefficients to obtain the target spectrum;

[0147] The spectrum conversion module 105 is used to perform inverse Fourier transform on the target spectrum to obtain the target audio with changed timbre.

[0148] Using the device provided in the embodiment of the present application, initial audio is obtained, and an initial spectrum of the initial audio and a spectrum envelope corresponding to the timbre are extracted; the spectrum envelope is scaled to obtain a target spectrum envelope; the target spectrum envelope and the spectrum envelope are used to obtain a weight coefficient; the initial spectrum is processed using the weight coefficient to obtain a target spectrum; the target spectrum is inverse Fourier transformed to obtain the target audio after the timbre is changed.

[0149] The initial audio is obtained, the initial spectrum of the initial audio and the spectrum envelope corresponding to the timbre are extracted, and the target spectrum envelope corresponding to the timbre change can be obtained by performing a stretching process on the spectrum envelope. Then, based on the spectrum envelope and the target spectrum envelope, a weight coefficient for processing the initial spectrum and realizing the timbre change is determined. The weight coefficient is applied to the initial spectrum to obtain the target spectrum. Finally, the target audio after the timbre change can be obtained by performing an inverse Fourier transform on the target spectrum. It can be seen that in the present application, the timbre change is not achieved by pitch change, but the spectrum envelope corresponding to the timbre is directly stretched. Since the spectrum envelope corresponding to the timbre is stretched and retracted, the pitch will not change synchronously, and thus the target audio obtained after the timbre change process will not have the problem of synchronous change in pitch. That is, the difference between the target audio after the timbre change and the initial audio is only the timbre change, but no pitch change, so the target audio will be more realistic.

[0150] In a specific implementation of the present application, the envelope processing module 102 is specifically used to obtain age simulation parameters from the client; determine the stretch coefficients corresponding to the age simulation parameters according to the negative correlation between the age simulation parameters and the stretch coefficients; and perform scaling processing on the spectrum envelope according to the stretch coefficients to obtain the target spectrum envelope.

[0151] In a specific implementation of the present application, the envelope processing module 102 is specifically used to obtain a reference age and a timbre simulation age corresponding to the initial audio; and determine an age simulation parameter based on a ratio of the reference age to the timbre simulation age.

[0152] In a specific implementation of the present application, the envelope processing module 102 is specifically used to stretch the spectrum envelope if the stretch coefficient is less than 1 to obtain a target spectrum envelope with a brighter timbre; if the stretch coefficient is greater than 1, compress the spectrum envelope to obtain a target spectrum envelope with a darker timbre.

[0153] In a specific implementation of the present application, the weight coefficient determination module 103 is specifically used to differentiate the target spectrum envelope and the spectrum envelope to obtain the initial mask spectrum coefficient; map the initial Mask spectrum coefficient to the linear domain, and use triangular window convolution to perform mask smoothing to obtain the weight coefficient.

[0154] In a specific implementation of the present application, the weight coefficient determination module 103 is specifically used to differentiate the target spectrum envelope and the spectrum envelope when the valid data interval of the target spectrum envelope is within the data interval of the spectrum envelope, and determine the differential value as the initial mask spectrum coefficient; if the valid data interval of the target spectrum envelope exceeds the data interval of the spectrum envelope, the corresponding initial mask spectrum coefficient is determined to be zero.

[0155] In a specific implementation of the present application, the spectrum processing module 104 is specifically configured to multiply the weight coefficient by the initial spectrum to obtain a target spectrum.

[0156] In a specific implementation of the present application, the spectrum conversion module 105 is specifically used to supplement the target spectrum with the corresponding Hening window weight coefficients; perform short-time Fourier inverse transform on the target spectrum supplemented with the Hening window weight coefficients to obtain the modulation of each frame; and perform overlap-addition on the modulation of each frame to obtain the target audio.

[0157] Corresponding to the above method embodiment, an embodiment of the present application further provides an electronic device. The electronic device described below and the audio processing method described above can refer to each other.

[0158] See also Figure 3 As shown, the electronic device includes:

[0159] A memory 332, for storing computer programs;

[0160] The processor 322 is used to implement the steps of the audio processing method of the above method embodiment when executing the computer program.

[0161] For details, please refer to Figure 4 , Figure 4 A specific structural diagram of an electronic device provided for this embodiment, the electronic device may have relatively large differences due to different configurations or performances, and may include one or more processors (central processing units, CPU) 322 (for example, one or more processors) and a memory 332, and the memory 332 stores one or more computer applications 342 or data 344. Among them, the memory 332 can be a temporary storage or a permanent storage. The program stored in the memory 332 may include one or more modules (not shown in the figure), and each module may include a series of instruction operations in the data processing device. Furthermore, the central processing unit 322 can be configured to communicate with the memory 332 to execute a series of instruction operations in the memory 332 on the electronic device 301.

[0162] The electronic device 301 may further include one or more power supplies 326 , one or more wired or wireless network interfaces 350 , one or more input and output interfaces 358 , and / or one or more operating systems 341 .

[0163] The steps in the audio processing method described above can be implemented by the structure of an electronic device.

[0164] Corresponding to the above method embodiment, the embodiment of the present application further provides a readable storage medium. The readable storage medium described below and the audio processing method described above can refer to each other.

[0165] A readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the audio processing method of the above method embodiment are implemented.

[0166] The readable storage medium may specifically be a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, or other readable storage medium that can store program codes.

[0167] Those skilled in the art may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented with electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in terms of function in the above description. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art may use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

Claims

1. An audio processing method, characterized in that: include: Acquire initial audio, and extract an initial spectrum of the initial audio and a spectrum envelope corresponding to the timbre; Performing scaling processing on the spectrum envelope to obtain a target spectrum envelope; Obtaining a weight coefficient using the target spectrum envelope and the spectrum envelope; Using the weight coefficients, processing the initial spectrum to obtain a target spectrum; Performing inverse Fourier transform on the target frequency spectrum to obtain target audio with changed timbre; Among them, the spectrum envelope corresponding to the timbre in the initial audio is extracted: Extract the baseband synchronization sequence of the initial audio, divide the baseband synchronization sequence into frames, and add a Hening window to obtain a segmented frame signal sequence; Perform short-time Fourier transform on the frame signal sequence to obtain the spectrum distribution of the initial audio; Using the frequency spectrum distribution, the power spectrum curve is obtained; Extracting the signal envelope of the power spectrum curve, and using the signal envelope to determine the spectrum envelope; The step of performing scaling processing on the spectrum envelope to obtain a target spectrum envelope includes: Get age simulation parameters from the client; According to the negative correlation between the age simulation parameter and the stretch-shrink coefficient, determining the stretch-shrink coefficient corresponding to the age simulation parameter; According to the stretch coefficient, the spectrum envelope is stretched and contracted to obtain the target spectrum envelope; Wherein, the step of obtaining age simulation parameters from the client includes: Obtaining a reference age and a timbre simulation age corresponding to the initial audio; wherein, after extracting the timbre feature of the initial audio, the timbre feature is identified to determine the reference age; The age simulation parameter is determined based on a ratio of the reference age to the timbre simulation age.

2. The audio processing method according to claim 1, characterized in that: The method of performing scaling processing on the spectrum envelope according to the scaling coefficient to obtain the target spectrum envelope includes: If the stretch coefficient is less than 1, stretching the spectrum envelope to obtain the target spectrum envelope with brighter timbre; If the stretch coefficient is greater than 1, the spectrum envelope is compressed to obtain the target spectrum envelope with a darker timbre.

3. The audio processing method according to claim 1, characterized in that: The target spectrum envelope and the spectrum envelope are used to obtain weight coefficients, including: Differentiating the target spectrum envelope and the spectrum envelope to obtain initial mask spectrum coefficients; The initial Mask spectrum coefficients are mapped to a linear domain, and mask smoothing is performed using a triangular window convolution to obtain the weight coefficients.

4. The audio processing method according to claim 3, characterized in that: The target spectrum envelope and the spectrum envelope are differentiated to obtain initial mask spectrum coefficients, including: If the valid data interval of the target spectrum envelope is within the data interval of the spectrum envelope, the target spectrum envelope and the spectrum envelope are differentiated, and the differential value is determined as the initial mask spectrum coefficient; If the valid data interval of the target spectrum envelope exceeds the data interval of the spectrum envelope, the corresponding initial mask spectrum coefficient is determined to be zero.

5. The audio processing method according to claim 1, characterized in that: The initial spectrum is processed by using the weight coefficient to obtain a target spectrum, including: The weight coefficient is multiplied by the initial spectrum to obtain the target spectrum.

6. The audio processing method according to claim 1, characterized in that: Performing an inverse Fourier transform on the target spectrum to obtain a target audio with a changed timbre, including: Supplementing the target spectrum with corresponding Hening window weight coefficients; Performing an inverse short-time Fourier transform on the target spectrum supplemented with the Henin window weight coefficients to obtain modulation of each frame; The pitch changes of the frames are overlapped and added to obtain the target audio.

7. An electronic device, characterized in that: include: Memory for storing computer programs; A processor, configured to implement the steps of the audio processing method according to any one of claims 1 to 6 when executing the computer program.

8. A readable storage medium, characterized in that: The readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the audio processing method according to any one of claims 1 to 6 are implemented.

Images