A method for broadcast language recognition based on pitch normalization reconstruction

Through the normalized reconstruction method of fundamental pitch, endpoint detection and all-pole filter model are used to solve the problem of low language recognition accuracy, achieving higher language recognition accuracy and natural speech reconstruction effect.

CN114913844BActive Publication Date: 2024-11-22KUNMING UNIV OF SCI & TECH
View PDF 10 Cites 0 Cited by

Patent Information

Application Number
CN202210376131.4
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2022-04-11
Publication Date
2024-11-22
Estimated Expiration
2042-04-11

AI Technical Summary

Technical Problem

In the real environment, the accuracy of language recognition is difficult to improve, the existing technology cannot effectively distinguish the deep-level characteristics of different languages, and the voice reconstruction effect is not natural enough.

Method used

The method of normalized reconstruction of the fundamental tone is used to extract the mute segment and the audio segment through endpoint detection, extract the fundamental tone frequency and normalize it, and convert it into a gate pulse using a voltage-controlled oscillator and differential judgment, and combine it with the full pole filter model to resonate. Construct audio signals to reduce the influence of speaker characteristics and improve language recognition accuracy.

Benefits of technology

It effectively reduces the impact of announcer's pronunciation characteristics on language recognition, improves the accuracy of broadcast audio language recognition, and preserves the complete semantics of speech in speech reconstruction.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN114913844B_ABST
    Figure CN114913844B_ABST
Patent Text Reader

Abstract

The present invention relates to a broadcast language recognition method of pitch normalization reconstruction, belonging to the technical field of language recognition. The present invention first performs endpoint detection on a speech signal to extract a silent segment and a voiced segment, then extracts the pitch frequency and the channel response in the voiced segment, and then normalizes the pitch frequency and converts it into a glottal pulse of the voiced segment through a voltage-controlled oscillator and differential judgment. The glottal pulse of the silent segment is composed of white noise, and the two are reconstructed through a full-pole filter model to obtain a normalized audio signal. The present invention can reduce the influence of the announcer's pronunciation characteristics on language recognition, thereby improving the accuracy of broadcast audio language recognition.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The invention relates to a broadcast language recognition method based on pitch normalization reconstruction, belonging to the technical field of language recognition. Background Art

[0002] With the continuous progress of society, language recognition technology is widely used in people's daily lives. There are more than 7,099 existing languages ​​​​in the world, and its application value is increasingly valued.

[0003] Broadcast interference and signal tampering often occur in border areas, and humans can master relatively few languages, so it is difficult to manually distinguish between languages ​​and determine whether signal transmission is normal.

[0004] The core issue of language recognition technology is how to extract the features that distinguish different languages. Traditional underlying acoustic features include Mel Frequency Cepstral Coefficient (MFCC), Gammatone Frequency Cepstrum Coefficient (GFCC), Perceptual Linear Predictive Coefficient (PLP), and LogMel-scale Filter Bank Energies (Fbank).

[0005] These underlying acoustic features have been widely used in language recognition, speaker recognition, and emotion recognition. However, many differences between languages ​​are reflected in prosodic features such as pronunciation, syntax, and semantics. However, in language recognition training, it cannot be guaranteed that the extracted acoustic features will not be affected by speaker characteristics and emotional characteristics, and whether these features are deep-level features that distinguish different languages ​​also needs further research.

[0006] As a branch of speech synthesis, speech reconstruction has common speech synthesis methods, such as waveform splicing, parameter synthesis, rule synthesis and deep learning. The waveform splicing method uses the materials in the speech material library to extract and splice the required speech, but the required materials are too large for different languages; the parameter synthesis method uses parameters such as fundamental frequency, formant, and vocal tract characteristics to synthesize speech. The vocoder is the most basic part of parameter synthesis. The traditional vocoder extracts acoustic and linguistic features as control conditions through preprocessing to synthesize the required speech, but the synthesis effect is not natural enough. The rule synthesis method uses phonemes to form syllables, and then integrates the rhythmic features of the language to synthesize speech. The synthesis is natural and clear, but the rhythmic features are not easy to extract. Summary of the invention

[0007] The technical problem to be solved by the present invention is to provide a broadcast language recognition method based on pitch normalization reconstruction, so as to solve the problem of difficulty in improving the accuracy of language recognition in a real environment.

[0008] The technical solution of the present invention is: a broadcast language recognition method of pitch normalization reconstruction, firstly, endpoint detection is performed on a speech signal to extract a silent segment and a voiced segment, then the pitch frequency and the channel response in the voiced segment are extracted, and then the pitch frequency is normalized and converted into a glottal pulse of the voiced segment through a voltage-controlled oscillator and differential judgment, the glottal pulse of the silent segment is composed of white noise, and the two are reconstructed through a full-pole filter model to obtain a normalized audio signal.

[0009] The specific steps are:

[0010] Step 1: Perform endpoint detection on the speech signal. Extract the subband entropy spectrum features and the one-dimensional features of the gammatone frequency cepstral coefficients from the speech signal, perform adaptive weighted fusion, cluster the fused features to obtain the decision threshold, and mark the voiced and silent segments in the speech signal according to the decision threshold value.

[0011] Step 2: Extract the fundamental frequency from the marked voiced segment and normalize it to the specified frequency range to construct a normalized glottal pulse excitation. The glottal pulse excitation of the silent segment is composed of white noise.

[0012] Step 3: Perform Fourier transform on the framed and windowed audio signal to obtain the logarithmic energy spectrum, then perform inverse Fourier transform and take the real part, and then take out the part with the most concentrated energy to construct the channel response.

[0013] Step 4: The glottal pulse excitation and vocal tract response are passed through the full-pole model and the high-frequency part of the signal is emphasized to reconstruct the speech with normalized fundamental frequency.

[0014] Step 5: Reconstruct all the speech in the corpus according to Step 1-Step 4 to obtain a normalized corpus, and then extract acoustic features from the audio of the corpus as distinguishing features between languages, and send them to the classification model for training and recognition, so as to obtain the language type to which the speech to be identified belongs.

[0015] The Step 1 is specifically as follows:

[0016] Step 1.1: The audio signal in the material library is framed and windowed, where the window length is 256 and the frame shift is 128. The energy spectrum is obtained by fast Fourier transform calculation of the short-time speech frame signal after windowing, as shown in formula (1):

[0017] E i (k) = |X i (k) 2(1)

[0018] Step 1.2: Divide each frame signal into N b sub-bands, each sub-band contains 4 spectral lines, calculate the probability of each sub-band energy:

[0019]

[0020] Step 1.3: Calculate the subband spectral entropy of each frame of speech signal:

[0021]

[0022] Step 1.4: Substitute the spectral line energy E obtained from equation (1) i (k) GFCC0 extraction through Gammatone filter:

[0023]

[0024] In the formula, H m (k) is the Gammatone filter response, m is the filter number, α is the exponential compression value, and the present invention takes

[0025] Then the GFCC features are obtained through discrete cosine transform:

[0026]

[0027] Among them, the 13-dimensional GFCC feature is required, so n = 13, and then the first dimension coefficient is extracted to obtain the GFCC0 feature G i .

[0028] Step 1.5: Perform median smoothing on the extracted two types of features, then perform amplitude translation adjustment and take the absolute value to obtain H i ′,G i ′, after obtaining the weight coefficient according to the average adaptive calculation, feature fusion is performed:

[0029] F i =α1H i ′+α2G i ′ (6)

[0030] Among them, α1 is the weight coefficient of the subband entropy spectrum feature, and α2 is the weight coefficient of the GFCC0 feature.

[0031] F i Normalize to get the fused eigenvalue:

[0032]

[0033] Step 1.6: Adaptively cluster the fusion feature F′ using the fuzzy C-means clustering algorithm, set the number of cluster centers to 2, and iteratively calculate the speech cluster center d voice and the noise cluster center d noice .

[0034] Step 1.7: Set the threshold value according to the cluster center:

[0035]

[0036] Among them, T h is the high threshold, T l is the low threshold value, β1 and β2 are empirical constants.

[0037] Step 1.8: Based on the double threshold value, the value below the threshold is represented by 0, and the frame is considered to be a silent segment. The value above the threshold is represented by 1, and the frame is considered to be a sound segment. The detection results of each frame are combined into an endpoint detection result vector v:

[0038] v=[v1,v2,v3,…,v i ],v i ∈{0,1} (9)

[0039] Among them, v i is the endpoint detection result of the i-th frame.

[0040] The Step 2 is specifically as follows:

[0041] Step 2.1: Silence and noise in the audio will affect the accuracy of pitch frequency estimation. Endpoint detection can distinguish between voiced and silent segments. Extracting the pitch frequency from the voiced segment can reduce the probability of estimation error.

[0042] If the VAD result of the i-th frame in v is 1, extract the voiced segment x of the i-th frame i (n) After Fourier transform with 256 points, we get signal X i (k), and then taking the logarithm, we get the logarithmic amplitude spectrum:

[0043] L i (k) = 20lg (X i (k)), 0<k≤256 (10)

[0044] Step 2.2: L i After inverse Fourier transform, the cepstrum sequence N of the current frame speech signal is obtained i, the harmonic peaks are shown to be equally spaced in the cepstrum, and the spacing is the fundamental frequency. The quotient of the sampling frequency and the fundamental frequency is the fundamental frequency. The present invention defaults to finding the maximum harmonic peak between 50 and 400 Hz, and the corresponding interval range is shown in S1 and S2 in formula (12):

[0045]

[0046] For sampling frequency f s = 8000Hz speech, only the maximum value is found in the interval 20 to 160 in the cepstrum sequence, that is:

[0047]

[0048] The fundamental frequency f1,…,f of each frame in the tth voiced segment L Construct the fundamental frequency vector f bm :

[0049]

[0050] Where l is the number of frames in a sound segment.

[0051] Step 2.3: In order to keep the speaker’s voice unchanged while changing the fundamental frequency, find The average value of each element f a , given the normalized fundamental frequency f c , normalize the fundamental frequency according to the following formula:

[0052]

[0053] Get the normalized fundamental frequency in the t-th voiced segment:

[0054]

[0055] Loop extraction and calculation to obtain the normalized fundamental frequency of a speech:

[0056]

[0057] Where T is the number of voiced segments.

[0058] The normalized fundamental frequency matrix Z is subjected to cubic spline interpolation to make the transition of the fundamental frequency between frames smoother, and then it is subjected to median filtering and linear smoothing to remove the wild points with estimation errors in the voiced segment to obtain the smoothed fundamental frequency Z1. Z1 is input into the voltage-controlled oscillator to generate the glottal pulse excitation signal Z2.

[0059] Step 2.4: If the VAD result of the i-th frame in v is 0, a Gaussian white noise Z5 with a frame length of 256 and a mean of 0 is generated

[0060] Delay Z2 by one unit to get matrix Z3, and subtract the two in turn to get a frame of glottal pulse excitation matrix Z4

[0061] The Step 3 is specifically as follows:

[0062] Step 3.1: Take the first 128 data of the logarithmic amplitude spectrum extracted from equation (11) to obtain the matrix P1:

[0063] P1=L i (k),0≤k≤128 (17)

[0064] Step 3.2: In order to separate the glottal excitation from the vocal tract excitation, P1 is subjected to discrete cosine transform to obtain the matrix P2. Because the two are in different cepstrum intervals in the cepstrum domain, the cepstrum is bounded by λ=25 spectral lines in the cepstrum, and the data in the interval 1 to 25 is constructed into the cepstrum matrix G of the vocal tract response, as shown in the following formula:

[0065]

[0066] Step 3.3: After taking the antilogarithm of G, we get G1 as shown in formula (19), and then flip it with G1 to get G2. G1 and G2 will be merged into G4, as shown in formula (19).

[0067]

[0068] G4={G2,G1} (20)

[0069] In order to reduce the amount of calculation, the part of the channel response parameters with relatively concentrated energy near the highest energy peak 128 is taken as the channel response parameter matrix. That is, the response points between [107,151] are taken to construct a new channel response parameter matrix.

[0070] The Step 4 is specifically as follows:

[0071] The reconstruction is performed frame by frame as the number of speech frames increases. When the current frame is a silent segment, the glottal impulse is generated by white noise Z5. The extracted glottal pulse excitation matrix G6 is used to put the glottal pulse excitation and vocal tract response parameters of the current frame into a full-pole filter to reconstruct the current frame speech W i

[0072] When the current frame is a voiced segment, the glottal pulse excitation matrix Z4 is extracted to put the glottal pulse excitation and the vocal tract response of the current frame into the full-pole filter to reconstruct the voiced segment current frame speech W i .

[0073] The filter parameters are updated once per frame, and finally the reconstructed speech of each frame in the voiced segment and the silent segment is superimposed to obtain the reconstructed speech:

[0074] W=[W1,W2,…,W i ] (twenty one)

[0075] Since the vocal tract model is reconstructed by an all-pole filter, the low-frequency part of the reconstructed speech is weakened, so amplitude normalization and pre-emphasis are required to improve the low-frequency part of the speech.

[0076] The beneficial effect of the present invention is that it can reduce the influence of the announcer's pronunciation characteristics on language recognition, thereby improving the accuracy of broadcast audio language recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0077] Figure 1 It is the overall structural diagram of the present invention;

[0078] Figure 2 It is an endpoint detection effect diagram of the present invention;

[0079] Figure 3 is a vocal tract response waveform diagram of a frame of speech signal of the present invention;

[0080] Figure 4 It is a frame of glottal impulse response and reconstructed speech waveform diagram of the present invention;

[0081] Figure 5 It is the original speech waveform diagram and spectrogram of the present invention;

[0082] Figure 6 It is the reconstructed original speech waveform diagram and spectrogram of the present invention. DETAILED DESCRIPTION

[0083] The present invention will be further described below in conjunction with the accompanying drawings and specific implementation methods.

[0084] Embodiment 1, as Figure 1 As shown, a method for broadcast language recognition based on pitch normalization reconstruction is described, and the specific steps are as follows:

[0085] Step 1: Test audio data acquisition

[0086] The data set is obtained from programs of international radio stations at different time periods, including six languages: Chinese, Tibetan, Lao, Cambodian, and Burmese. The broadcast audio of each language is mono with a sampling rate of 8000Hz, a bit number of 16, and a duration of 10 seconds. The background music noise and telephone interview noise interference during audio playback were manually removed.

[0087] Step 2: Voice endpoint detection

[0088] The endpoint detection is performed according to the endpoint detection method of the present invention, and the endpoint detection results are shown in Figure 2 ,From the figure, we can see that a 10-second speech is marked into several segments, among which the voiced segment only contains speech waveforms, and the silent segment does not contain speech waveforms.

[0089] Step 3: Pitch frequency extraction and normalization

[0090] Extract the voiced segment x of the i-th frame with the endpoint detection result of 1 i (n) After Fourier transform with 256 points, we get signal X i (k), take the logarithm and get the logarithmic amplitude spectrum L l .

[0091] For L l After performing inverse Fourier transform, the maximum harmonic peak value between 50 and 400 Hz of the fundamental frequency is found, that is, the maximum value is found in the range of 20 to 160.

[0092] The fundamental frequency of each frame in all voiced segments is extracted cyclically to obtain the fundamental frequency sequence F = [f1, f2, f3, ..., f i ]

[0093] According to formula (14), the extracted fundamental frequency is normalized to ensure that the speech tone does not change too much after normalization.

[0094] The normalized fundamental frequency matrix F is subjected to cubic spline interpolation to make the matrix smoother, and then the matrix Z2 is obtained through median filtering and linear smoothing.

[0095] Input Z2 into the voltage-controlled oscillator and perform differential judgment to obtain the glottal pulse excitation Z3, such as Figure 4 shown.

[0096] Step 4: Vocal channel parameter extraction

[0097] The first 128 data of the extracted logarithmic amplitude spectrum are taken out and discrete cosine transform is performed to obtain the cepstrum domain.

[0098] The cepstrum matrix G of the vocal tract impulse response is constructed by taking the λ=25 spectral lines in the cepstrum as the boundary and the data in the interval 1 to 25.

[0099] Taking the antilogarithm of G yields G1.

[0100] Then flip it with G1 to get G2, and G1 and G2 will be merged into G4

[0101] Then perform inverse Fourier transform on G4 and take the real part, and then take out the data between [107,151] where the energy is relatively concentrated, and obtain the full-pole filter matrix G5 of the vocal tract model. Figure 3 shown.

[0102] Step 5: Voice reconstruction

[0103] The reconstruction is performed frame by frame as the number of speech frames increases. When the current frame is a silent segment, the glottal impulse is Z5 generated by white noise.

[0104] When the current frame is a voiced segment, the glottal pulse excitation matrix G5 is extracted to put the glottal pulse excitation and vocal tract response parameters of the current frame into the full-pole filter to reconstruct the current frame speech W in the voiced segment. i When the current frame is a silent segment, the glottal pulse excitation composed of white noise in the current frame is put into the vocal tract response into the full-pole filter to reconstruct the speech W of the current frame in the silent segment. i , the synthesis effect of a frame of speech is as follows Figure 4 shown.

[0105] The filter parameters are updated once per frame, and finally the reconstructed speech of each frame is superimposed to obtain the reconstructed speech.

[0106] After amplitude normalization and pre-emphasis to improve the high frequency part of the speech, the waveform and spectrogram of the original speech are as follows Figure 5 As shown, the waveform and spectrogram of the reconstructed speech are as follows Figure 6 shown.

[0107] As can be seen from the figure, the reconstructed speech signal is not much different from the original speech and can retain the complete semantics of the speech.

[0108] Step 6: Language Identification

[0109] The present invention adopts the Gaussian mixture general background model as the language recognition training model. 300 speech sounds are randomly selected from the data set of each language as the GMM training set, and the 6 languages ​​have a total of 1800 training corpora. The UBM training set is 300 speech sounds randomly selected from the data set as the training corpus, and the 6 languages ​​have a total of 1800 speech sounds. 171 speech sounds of each language are randomly selected from the data set as the test set, and the 6 languages ​​have a total of 1026 speech sounds.

[0110] MFCC, GFCC, and PLP features are extracted as distinguishing features for language recognition. The reconstruction model is used to normalize the fundamental frequency of the test set and the training set. The normalization range is normalized to 100Hz~300Hz for model training and recognition. The recognition results are shown in Table 1.

[0111]

[0112] Table 1: Average recognition rate of languages ​​with pitch frequency normalized to different ranges (%)

[0113] As can be seen from Table 1, the average language recognition accuracy of the extracted underlying acoustic features has been improved after speech reconstruction. Secondly, no matter which frequency range the fundamental frequency is normalized to, it will not have a particularly large impact on language recognition, which also shows that when extracting the underlying acoustic features, changes in the fundamental frequency will not change the feature vectors that distinguish languages.

[0114] The specific implementation modes of the present invention are described in detail above in conjunction with the accompanying drawings, but the present invention is not limited to the above implementation modes, and various changes can be made within the knowledge scope of ordinary technicians in this field without departing from the purpose of the present invention.

Claims

1. A method for broadcast language recognition based on pitch normalization reconstruction, characterized in that: Step 1: Perform endpoint detection on the speech signal. Extract subband entropy spectrum features and gammatone frequency cepstral coefficient one-dimensional features from the speech signal and perform adaptive weighted fusion. Cluster the fused features to obtain the decision threshold. Mark the voiced and silent segments in the speech signal according to the decision threshold value. Step 2: Extract the fundamental frequency from the marked voiced segment and normalize it to the specified frequency range to construct a normalized glottal pulse excitation. The glottal pulse excitation of the silent segment is composed of white noise. The method of constructing a normalized glottal pulse is specifically as follows: Among them, f bm represents the fundamental frequency vector, l is the number of frames in a voiced segment, f1,…,f L Indicates the fundamental frequency of each frame in the sound segment, and t indicates the number of the sound segment; Step 3: Perform Fourier transform on the framed and windowed audio signal to obtain the logarithmic energy spectrum, then perform inverse Fourier transform and take the real part, then take out the part with the most concentrated energy to construct the channel response; Step 4: The glottal pulse excitation and vocal tract response are passed through the full-pole model and the high-frequency part of the signal is emphasized to reconstruct the speech with normalized fundamental frequency; Step 5: Reconstruct all the speech in the corpus according to Step 1-Step 4 to obtain a normalized corpus, then extract acoustic features from the audio of the corpus as distinguishing features between languages, and send them to the classification model for training and recognition, so as to obtain the language type to which the speech to be identified belongs; The Step 1 is specifically as follows: Step 1.1: The audio signal in the material library is framed and windowed, where the window length is 256 and the frame shift is 128. The energy spectrum is obtained by fast Fourier transform calculation of the short-time speech frame signal after windowing, as shown in formula (1): E i (k)=|X i (k) 2 (1) Step 1.2: Divide each frame signal into N b sub-bands, each sub-band contains 4 spectral lines, calculate the probability of each sub-band energy: Step 1.3: Calculate the subband spectral entropy of each frame of speech signal: Step 1.4: Substitute the spectral line energy E obtained from equation (1) i (k) GFCC0 extraction through Gammatone filter: In the formula, H m (k) is the Gammatone filter response, m is the filter number, and α is the exponential compression value; Then the GFCC features are obtained through discrete cosine transform: Extract the first dimension coefficient to obtain GFCC0 feature G i ; Step 1.5: Perform median smoothing on the extracted two types of features, then perform amplitude translation adjustment and take the absolute value to obtain H i ′,G i ′, after obtaining the weight coefficient according to the average adaptive calculation, feature fusion is performed: F i =α1H i ′+α2G i ′ (6) Among them, α1 is the weight coefficient of the subband entropy spectrum feature, and α2 is the weight coefficient of the GFCC0 feature; F i Normalize to get the fused eigenvalue: Step 1.6: Adaptively cluster the fusion feature F′ using the fuzzy C-means clustering algorithm, set the number of cluster centers to 2, and iteratively calculate the speech cluster center d voice and the noise cluster center d noice ; Step 1.7: Set the threshold value according to the cluster center: Among them, T h is the high threshold, T l is the low threshold value, β1 and β2 are empirical constants; Step 1.8: Based on the double threshold value, the value below the threshold is represented by 0, and the i-th frame is considered to be a silent segment. The value above the threshold is represented by 1, and the i-th frame is considered to be a sound segment. The detection results of each frame are combined into an endpoint detection result vector v: v=[v1,v2,v3,…,v i ],v i ∈{0,1} (9) Among them, v i is the endpoint detection result of the i-th frame.

Citation Information

Patent Citations

  • Method for forming the excitation signal for a glottal pulse model based parametric speech synthesis system

    CA3178027A1

  • Vocoder based on residual signal spectrum reconfiguration

    CN102750955A

  • Phonetic feature processing method of voiceprint identification in noise environment

    CN105679312A

  • Voice background noise recognition method based on endpoint detection

    CN112837704A

  • Language identification method based on Gaussian weighted GFCC features

    CN114093343A