A sparse decomposition based method for sibilance suppression

Abstract

The present application belongs to the technical field of digital audio processing, and specifically provides a sibilance suppression method based on sparse decomposition, to solve the problem that the existing sibilance suppression method cannot distinguish sibilance and useful speech in fine granularity, resulting in perceptible loss of useful speech information when suppressing sibilance. The present application includes two stages of sibilance dictionary construction and sibilance suppression. The method extracts the spectrum of the training sibilance audio frame, takes the spectral amplitude data of the sibilance frequency band as training data, uses a sparse dictionary training method to train and generate a sibilance dictionary, and then performs sparse decomposition on the extracted test audio data in units of frames on the dictionary. The number of decomposition atoms is determined by using the energy of the residual vector, and the sibilance and useful speech are distinguished in fine granularity according to the size of the decomposition coefficient. At the same time, the sibilance component is removed by retaining the residual signal, so as to effectively suppress the sibilance in the audio while maintaining the original speech information.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of digital audio processing, and specifically provides a sibilance suppression method based on sparse decomposition. BACKGROUND

[0002] Sibilance is the sound produced by airflow passing through the teeth when the tip of the tongue is pressed against the upper front teeth while pronouncing. Generally, people are prone to sibilance when pronouncing words such as 'zhi', 'ci' and'si'. Because the frequency of sibilance is relatively high, it sounds sharp, so sibilance in audio may affect the listening experience of the audience. After professional recording of songs or dubbing, the recorded audio usually needs to be edited to improve the listening effect, and sibilance suppression is one of the processes.

[0003] Sibilance suppression can be achieved through frequency filtering (such as EQ equalizer), and some audio processing software (such as Adobe Audition) has the function of sibilance suppression, but the technology is not disclosed. There are few reports on sibilance suppression methods in the literature. In the disclosed technology, the invention patent "202110469577.7, audio signal processing method and device" attenuates sibilance by dynamically compressing the sibilant frequency segment. Since the sibilant frequency segment also contains speech components, this method uniformly processes the audio containing speech and sibilance without distinction, which may result in loss of speech information. The invention patent "202210225928.4, a dubbing audio analysis method, device, electronic equipment and storage medium" aims to improve the quality of audio containing abnormal sounds (such as saliva sound, sibilance, bubble sound and breathing sound, etc.). According to the current sound spectrum image of the audio to be processed and the sound spectrum image of the pre-classified problematic audio, image recognition technology and algorithm clustering analysis technology are used to determine whether the dubbing audio has pronunciation abnormalities and determine the abnormal type. Finally, the frequency, amplitude, timbre, pitch and loudness of the sound are corrected to correct the dubbing audio. This method identifies multiple types of saliva sound, sibilance, bubble sound and breathing sound through image recognition technology. Since the sound spectrum image features of these abnormal types have similar components, the classification accuracy is not good enough. In addition, like the previous method, this method also modifies the entire audio, inevitably damaging the original speech information.

[0004] In summary, the current sibilance suppression method does not distinguish between sibilance and useful speech in a fine-grained manner, resulting in perceptible loss of useful speech information when suppressing sibilance. SUMMARY

[0005] The purpose of the present application is to provide a more effective sibilance suppression method based on sparse decomposition to solve the problem of perceptible loss of useful speech information when suppressing sibilance due to the lack of fine-grained distinction between sibilance and useful speech in the current method.

[0006] To achieve the above object, the technical scheme of the present application is as follows:

[0007] A sparse decomposition-based sibilance suppression method, characterized in that it comprises two stages of sibilance dictionary construction and sibilance suppression.

[0008] The sibilance dictionary construction comprises the following steps:

[0009] A1. Generating sibilance samples;

[0010] A1-1. Initializing a sibilance feature matrix F as empty;

[0011] A1-2. Generating the sibilance feature matrix F according to sibilance training audio, for each sibilance training audio:

[0012] A1-2-1. Dividing the audio data into audio frames of a fixed length with overlapping between adjacent frames to obtain N audio frames; and performing windowing processing on the audio frame data and then performing short-time Fourier transform to obtain the corresponding frequency spectrum;

[0013] A1-2-2. Assuming that there are M+1 data corresponding to each time in the frequency spectrum amplitude data, then the sampling frequency points m 5000 and m 10000 are calculated respectively:

[0014]

[0015]

[0016] wherein, is a down-round operation, f max is the highest frequency;

[0017] A1-2-3. For each audio frame: for the nth, 1≤n≤N, audio frame, the amplitude data of the sampling frequency points m 5000 to m 10000 are arranged to form a column vector and normalized to obtain a feature vector f n , and f n is appended to the sibilance feature matrix F by column;

[0018] A2. Constructing a sibilance dictionary: setting the dictionary size as K, taking the feature vectors in the sibilance feature matrix F as training data, and using a sparse dictionary training method to train and generate a sibilance dictionary D;

[0019] The sibilance suppression comprises the following steps:

[0020] B1. Generating a test sample;

[0021] B1-1. Initialize the test matrix F' as empty;

[0022] B1-2. Generate the test matrix F' according to the audio to be processed, specifically:

[0023] B1-2-1. Divide the audio data to be processed into audio frames of a fixed length, with an overlap between adjacent frames, to obtain N' audio frames; and perform windowing processing on the audio frame data, and then perform a short-time Fourier transform to obtain the corresponding frequency spectrum, wherein the frequency spectrum amplitude data of the nth audio frame is e n ;

[0024] B1-2-2. For each audio frame, perform: 5000 10000 n n ;

[0025] B2. Sparse tooth sound decomposition;

[0026] B2-1. Define a category label array S with a length of N';

[0027] B2-2. For each audio frame, perform:

[0028] B2-2-1. Under the tooth sound dictionary D, perform sparse decomposition on the nth column vector f n of F' to obtain k atoms and a residual vector

[0029]

[0030] where d j and c j are the selected jth atom and its corresponding weight coefficient, respectively, and k is the smallest integer that satisfies the following formula:

[0031]

[0032] where and are the squared sum of each element of the residual vector and , respectively;

[0033] B2-2-2. Search for the maximum value c j in the k weight coefficients c max , 1≤j≤k, and assign the nth element S[n] of the category label array S according to c max :

[0034] ​​​

[0035] Where T is the preset weight coefficient threshold;

[0036] B3. Remove dental components:

[0037] B3-1. Initialize n = 1;

[0038] B3-2. Make a decision on S[n]:

[0039] If S[n] = 1, then update the spectral amplitude data e. n The mth 5000 From the mth 10000 One element:

[0040]

[0041] in, for The mm 5000 +1 element, ElementAbs(x) represents the operation of taking the absolute value of each element of vector x;

[0042] Otherwise, proceed to step B3-3;

[0043] B3-3. Update n to n+1, and determine whether n is greater than N′. If yes, go to step B3-4; otherwise, go to step B3-2.

[0044] B3-4. Perform a short-time inverse Fourier transform using the updated spectral data to obtain the processed audio.

[0045] Furthermore, the weighting coefficient threshold T has a range of values: 1≤T≤3.

[0046] Based on the above technical solution, the beneficial effects of the present invention are as follows:

[0047] This invention provides a sibilance suppression method based on sparse decomposition. The method extracts the spectrum of training sibilance audio frames, uses the spectral amplitude data of the sibilance frequency band as training data, and trains a sibilance dictionary using a sparse dictionary training method. On this dictionary, the extracted test audio data is sparsely decomposed frame by frame. The energy of the residual vector is used to determine the number of atoms in the decomposition, and the sibilance and useful speech are finely distinguished according to the magnitude of the decomposition coefficient. At the same time, the sibilance component is removed by retaining the residual signal, thereby effectively suppressing sibilance in the audio while maintaining the original speech information. Attached Figure Description

[0048] Figure 1 This is a flowchart illustrating the dentition suppression method based on sparse decomposition in this invention, which includes two stages: dentition dictionary construction and dentition suppression.

[0049] Figure 2 Figure for effect comparison of the sibilance suppression method based on sparse decomposition in the embodiment of the present application, wherein (a) is the spectral amplitude graph of the test audio, (b) is the spectral amplitude graph after sibilance processing by the audio processing software Adobe Audition, and (c) is the spectral amplitude graph after processing by the present application. DETAILED DESCRIPTION

[0050] In order to make the objectives, technical solutions and advantages of the present application clearer, further detailed description will be made to the present application in combination with the drawings and embodiments.

[0051] The embodiment provides a sibilance suppression method based on sparse decomposition, the flow of which is shown in Figure 1, and the method comprises two stages of sibilance dictionary construction and sibilance suppression. Figure 1

[0052] Firstly, the sibilance dictionary construction step will be further described as follows:

[0053] A1. Generating sibilance samples:

[0054] A1-1. Initializing the sibilance feature matrix F as empty;

[0055] A1-2. Generating the sibilance feature matrix F: suppose there is a group of sibilance audios for training, and for each of the audios, do the following:

[0056] In the embodiment, there are totally 50 sibilance audios for training, each of which is a single-channel audio with a sampling rate of 48000 Hz, and the following steps will be described by taking the processing of the first audio as an example;

[0057] A1-2-1. Dividing the audio data into audio frames according to fixed length, with overlapping between adjacent frames, and suppose the total number of the obtained audio frames is N; performing windowing processing on the audio frame data, and then performing short-time Fourier transform to obtain the corresponding spectrum;

[0058] In the embodiment, the audio data is equally divided into audio frames according to the length of 512 sampling points, with half-frame length overlapping between adjacent frames, and the total number of the obtained audio frames N = 8; the Hamming window is used for windowing processing;

[0059] A1-2-2. Suppose there are M+1 amplitude data in the spectrum amplitude data of an audio frame, corresponding to M+1 frequency points from 0 frequency to the highest frequency f max of the signal, and the following are calculated respectively:

[0060]

[0061] ​

[0062] in, This is a floor function;

[0063] In this embodiment, in the amplitude data of the spectrum, there are a total of M+1=257 amplitude data points at each moment of an audio frame, corresponding to frequencies from 0 to the highest frequency f of the signal. max The 257 frequency points between 24000Hz and 24000Hz are used to calculate the frequency point number m corresponding to 5000Hz using the formula. 5000 =54,10000Hz corresponds to the frequency point number m 10000 =107;

[0064] A1-2-3. For each n satisfying 1≤n≤N, perform the following: For the nth frame of audio, starting from m... 5000 to m 10000 Arrange the spectral amplitude data corresponding to the sampling frequency points into a vector. right Normalization will be performed, that is... Divide each element by The 2-norm is used to obtain the feature vector f of the nth frame. n , will f n The columns are appended to the dental sound feature matrix F, i.e., F = [F, f n ];

[0065] In this embodiment, N = 8, so for each n satisfying 1 ≤ n ≤ 8, a feature vector for the corresponding frame is generated; taking the audio of the n=1th frame as an example, starting from the frequency point numbered m 5000 =54 to frequency point number m 10000 =107, arrange the spectral amplitude data corresponding to these 54 sampling frequency points into a vector. have Its 2-norm is 14.673, for Normalization will be performed, that is... Dividing each element by its 2-norm (14.673) yields the feature vector f1 = [0.100, 0.052, ..., 0.029] for the n=1th frame. T Add f1 column by column to the dental sound feature matrix F, i.e., F = [F, f1];

[0066] A2. Constructing the dental consonant dictionary: Let K be the preset dictionary size, i.e. the number of atoms in the dictionary. Use the feature vectors in the dental consonant feature matrix F as training data, and use the sparse dictionary training method to train and generate the dental consonant dictionary D.

[0067] In the embodiment, the preset dictionary size K=256, the feature vector in the sibilance feature matrix F is taken as the training data, the sibilance dictionary D is generated by using the sparse dictionary training method K-SVD, wherein there are K=256 atoms in D, and each atom is a 54-dimensional column vector.

[0068] The sibilance suppression step is further described below:

[0069] B1. Generating a test sample:

[0070] B1-1. Initializing the test matrix F' as empty;

[0071] B1-2. Generating the test matrix F':

[0072] In the embodiment, the test audio is a single-channel audio with a sampling rate of 48000 Hz, the audio has a length of 6s, and contains 4 sibilance intervals. The first sibilance interval appears at 2.141s and has a duration of 0.064s, the second sibilance interval appears at 2.381s and has a duration of 0.160s, the third sibilance interval appears at 2.760s and has a duration of 0.026s, and the fourth sibilance interval appears at 4.660s and has a duration of 0.011s.

[0073] B1-2-1. Dividing the input test audio data into audio frames according to a fixed length, with overlapping between adjacent frames, and setting the total number of obtained audio frames as N'; performing windowing processing on each audio frame data, and then performing short-time Fourier transform to obtain the corresponding spectrum, and taking the vector formed by the spectrum amplitude data of the nth frame as e n .

[0074] In the embodiment, the audio data is equally divided into audio frames according to a length of 512 sampling points, with a length of half a frame of overlapping between adjacent frames, and the total number of obtained audio frames N'=1124; wherein the 401th to 413th frames are the first sibilance interval, the 446th to 476th frames are the second sibilance interval, the 517th to 522nd frames are the third sibilance interval, and the 873rd to 875th frames are the fourth sibilance interval; the Hamming window is used for windowing processing, and after the short-time Fourier transform, the spectrum corresponding to each audio frame is obtained, and the amplitude graph is shown in (a) of Figure 2 , taking the nth=401th frame as an example, the vector formed by the spectrum amplitude data of the frame e 401 =[1.649, 9.533, …, 0.021] T .

[0075] B1-2-2. For each n satisfying 1≤n≤N', performing: for the nth audio frame, performing: 5000 . 10000Arrange the spectral amplitude data corresponding to the sampling frequency points into a vector, denoted as f′. n f′ n The columns are appended to the test matrix F′, i.e., F′ = [F′, f′]. n ];

[0076] In this embodiment, N′=1124, so for each n satisfying 1≤n≤1124, a vector corresponding to the frame is generated: taking the audio of the n=401st frame as an example, starting from the frequency point numbered m 5000 =54 to frequency point number m 10000 =107, arrange the spectral amplitude data corresponding to these 54 sampling frequency points into a column vector f′. 401 =[0.314, 0.489, ..., 0.095] T f′ 401 The columns are appended to the test matrix F′, i.e., F′ = [F′, f′]. 401 ];

[0077] B2. Decomposition of sparse dental consonants:

[0078] B2-1. Define a category label array S of length N′;

[0079] B2-2. For each n satisfying 1≤n≤N′, perform:

[0080] In this embodiment, N′=1124, therefore for each n satisfying 1≤n≤1124, we perform the following:

[0081] B2-2-1. Under the dental consonant dictionary D, the nth column vector f′ of F′ n Sparse decomposition involves iteratively selecting k atoms from a dictionary D, one atom at a time, and then using linear combinations thereof. To represent f′ n The residual vector when selecting k atoms is obtained. for

[0082]

[0083] Where, d j and c j Let be the selected j-th atom and its corresponding weight coefficient, respectively, and k be the smallest integer satisfying the following formula:

[0084]

[0085] in, and These are the residual vectors. and The sum of squares of all elements;

[0086] In the embodiment, taking n=401 columns as an example, f′ 401 =[0.314, 0.489, …, 0.095] T Sparse decomposition is performed, when the number of selected atoms k=1, 2, 3, the sum of squares of each element of the residual vector is respectively There are Therefore, k=2 is the smallest integer satisfying , that is, the linear combination of two selected atoms is used to represent f′ 401 , wherein the first selected atom is d1=[0.158, 0.094, …, 0.028] T , the corresponding weight coefficient c1=5.037, the second selected atom is d2=[0.099, 0.052, …, 0.029] T , and the corresponding weight coefficient c2=1.326, at this time, the linear combination of the two atoms u 401 =[0.929, 0.543, …, 0.177] T , and the residual vector

[0087] B2-2-2. Search for the maximum value c j in k weight coefficients c max , 1≤j≤k, and assign a value to the nth element S[n] of the category label array S according to c max

[0088]

[0089] Wherein, T is a preset weight coefficient threshold;

[0090] In the embodiment, the value range of the weight coefficient threshold T is: 1≤T≤3, and the value of T is 2; taking n=401 frames as an example, since k=2, the maximum value c max =c1=5.037 in the weight coefficients c1 and c2 is found, and therefore c max ≥T, the nth=401 element S

[401] of the category label array S is assigned a value of 1;

[0091] B3. Remove the sibilance component:

[0092] B3-1. Initialize n=1;

[0093] B3-2. If S[n]=1, update the m n th to m 5000 th elements of the spectrum amplitude data vector e 10000 ​​

[0094]

[0095] wherein, is the mth element of the vector e 5000 +1, ElementAbs(x) represents an operation of taking absolute value of each element of the vector x;

[0096] Otherwise, go to step B3-3.

[0097] In this embodiment, take n=401 frames as an example. Since S

[401] =1, update the spectrum amplitude data vector e 401 +1, the mth element of the vector e 5000 +1, the mth element of the vector e 10000 +1:

[0098]

[0099] The 54th to 107th elements of e 401 +1 before updating are 0.314, 0.489, …, 0.095, respectively. After step B3-2, these elements are updated to 0.615, 0.055, …, 0.081, respectively.

[0100] B3-3. Update n to n+1, and determine whether n is greater than N', if yes, go to step B3-4; otherwise, go to step B3-2.

[0101] In this embodiment, N' = 1124, so update n to n+1, and determine whether n is greater than 1124, if yes, go to step B3-4; otherwise, go to step B3-2.

[0102] B3-4. Perform inverse short-time Fourier transform using the updated spectrum data to obtain the processed audio.

[0103] Next, the sibilance suppression effect of the method of the present application is compared with that of the audio processing software Adobe Audition for an embodiment, and the results are shown in Figure 2 Fig. 1; from Figure 2 Fig. 1(a), it can be seen that sibilance appears at four places of 2.141s, 2.381s, 2.760s and 4.660s in the original audio, and the corresponding spectrum amplitude graph has a high highlight in the frequency range of 5000Hz to 10000Hz; as shown in Figure 2 Fig. 1(b), after processing by Adobe Audition, the highlight is weakened, indicating that the sibilance component is suppressed; as shown in Figure 2 Fig. 1(c), after processing by the method of the present application, the sibilance suppression effect at the four places is more obvious, and there is no subjective feeling of loss of speech content.

[0104] The above description is only a specific implementation of the present application. Any feature disclosed in this specification, unless otherwise specifically stated, can be replaced by other equivalent or similar features. Any feature disclosed in this specification, or any method or process disclosed in this specification, can be combined with any other disclosed feature or method or process, unless otherwise specifically stated.

Claims

1. A method for suppressing dental consonants based on sparse decomposition, characterized in that, It includes two stages: dental consonant dictionary construction and dental consonant suppression; The construction of the dental consonant dictionary includes the following steps: A1. Generate dental sound samples; A1-1. Initialize the dental sound feature matrix F to be empty; A1-2. Generate the dentate feature matrix F based on the dentate training audio, and perform the following for each dentate training audio: A1-2-1. Divide the audio data into audio frames of fixed length, with overlap between adjacent frames, to obtain N audio frames; then perform windowing processing on the audio frame data, and then perform short-time Fourier transform to obtain its corresponding spectrum; A1-2-2. Suppose there are M+1 data points corresponding to each time step in the spectral amplitude data. Then calculate the sampling frequency point m respectively. 5000 With m 10000 : in, For floor operations, f max The highest frequency; A1-2-3. For each audio frame, perform the following: For the nth audio frame (1≤n≤N), set the sampling frequency point m... 5000 to m 10000 The amplitude data are arranged to form a column vector. And normalize to obtain the eigenvector f n , will f n Add the columns to the dental sound feature matrix F; A2. Constructing the dental consonant dictionary: Set the dictionary size to K, use the feature vectors in the dental consonant feature matrix F as training data, and use the sparse dictionary training method to train and generate the dental consonant dictionary D; The sibilance suppression includes the following steps: B1. Generate test samples; B1-1. Initialize the test matrix F′ to be empty; B1-2. Generate the test matrix F′ based on the audio to be processed, specifically as follows: B1-2-1. Divide the audio data to be processed into audio frames of fixed length, with overlap between adjacent frames, resulting in N′ audio frames. Window the data of each audio frame and then perform a short-time Fourier transform to obtain its corresponding spectrum. The amplitude of the spectrum of the nth audio frame is e. n ; B1-2-2. For each audio frame, perform the following: For the nth audio frame (1≤n≤N′), set the sampling frequency point m... 5000 to m 10000 The amplitude data are arranged to form a column vector f n ′, f n 'Append column by column to test matrix F'; B2. Decomposition of sparse dental consonants; B2-1. Define a category label array S of length N′; B2-2. Perform the following for each audio frame: B2-2-1. Under the dental consonant dictionary D, for the nth column vector f of F′ n Perform sparse decomposition to obtain k atoms and the residual vector. Where, d j and c j Let be the selected j-th atom and its corresponding weight coefficient, respectively, and k be the smallest integer satisfying the following formula: in, and These are the residual vectors. and The sum of squares of all elements; B2-2-2. Search for k weight coefficients c j The maximum value c in 1≤j≤k max According to c max Assign a value to the nth element S[n] of the category label array S: Where T is the preset weight coefficient threshold; B3. Remove dental components: B3-1. Initialize n = 1; B3-2. Make a decision on S[n]: If S[n] = 1, then update the spectral amplitude data e. n The mth 5000 From the mth 10000 One element: in, for The mm 5000 +1 element, ElementAbs(x) represents the operation of taking the absolute value of each element of vector x; Otherwise, proceed to step B3-3; B3-3. Update n to n+1, and determine if n is greater than N′. If yes, go to step B3-4; otherwise, go to step B3-2. B3-4. Perform a short-time inverse Fourier transform using the updated spectral data to obtain the processed audio.

2. The sibilant suppression method based on sparse decomposition as described in claim 1, characterized in that, The value range of the weighting coefficient threshold T is: 1≤T≤3.

Images