A real-time speech pitch contour extraction method and system

Abstract

This invention discloses a real-time speech fundamental frequency contour extraction method and system, comprising: preprocessing a received frame of sound signal, wherein the preprocessed signal includes several sampling points, each sampling point having a corresponding amplitude; performing local extremum detection on the preprocessed signal to obtain a set of maxima and a set of minima; selecting one of the maxima and minima sets as a selected extremum set based on the maxima and minima sets and a first preset model; filtering the selected extremum set according to a second preset model to obtain a filtered extremum set; and performing fundamental frequency F0 calculation and time-domain smoothing on the filtered extremum set. This invention achieves speech fundamental frequency estimation by directly detecting and filtering the local extremum positions of the time-domain waveform of the preprocessed sound signal, greatly reducing the computational overhead of real-time fundamental frequency detection without significantly affecting the calculation accuracy.

Description

Technical Field

[0001] This invention belongs to the field of speech signal processing technology, and specifically relates to a method and system for real-time speech fundamental frequency contour extraction. Background Technology

[0002] The fundamental frequency (FFF) of speech is one of the key parameters of speech signals, playing an important role in research fields such as tone recognition, melody recognition, speech synthesis, and speech coding. It is related to factors such as the characteristics of the speaker's vocal cords, pronunciation habits, gender, and age. Generally, the FFF range for human speech is approximately 100–400 Hz. Male speech typically has a lower FFF, generally between 100 and 200 Hz, while female and child speech typically has a higher FFF, generally between 200 and 400 Hz. FFF extraction is a fundamental problem in speech signal processing. The most classic traditional FFF detection algorithms are the Autocorrelation Function (ACF) method and the Average Magnitude Difference Function (AMDF) method, both based on time-domain signal analysis. The ACF method estimates the FFF by observing large peaks in the autocorrelation function curve of the speech signal at integer multiples of the FFF period. The AMDF method replaces the multiplication operation in the ACF method with a simpler difference operation, reducing computation time. Its principle is based on the existence of valleys in the AMDF function at integer multiples of the pitch period to estimate the pitch period. Besides time-domain methods, there are also pitch detection methods based on frequency-domain signal analysis, such as the overtone inner product spectrum method and cepstral analysis. Furthermore, there are hybrid methods that combine time-domain and frequency-domain analysis, avoiding detection errors caused by relying solely on time-domain or frequency-domain signal analysis. Additionally, there are machine learning-based methods and, more currently, cutting-edge deep neural network-based pitch extraction techniques.

[0003] Although there are many ways to achieve fundamental frequency extraction, real-time speech fundamental frequency extraction in ultra-low-power devices such as hearing aids and cochlear implants still presents difficulties and challenges, especially for cochlear implant systems, where the low-power requirements and control over the algorithm complexity are far greater than in hearing aids. Therefore, implementing a low-computational-power speech fundamental frequency detection method is particularly important for such devices. Summary of the Invention

[0004] To address the aforementioned issues, this invention provides a real-time speech fundamental frequency contour extraction method and system. By directly detecting and filtering the local extrema positions of the preprocessed audio signal time-domain waveform, the speech fundamental frequency is estimated. This significantly reduces the computational overhead of real-time fundamental frequency detection without significantly affecting the accuracy of fundamental frequency calculation, making it more suitable for operation on terminal devices with low computing power and low power consumption requirements.

[0005] The technical solution of the present invention is as follows: A real-time speech fundamental frequency contour extraction method, comprising the following steps:

[0006] The received frame of sound signal is preprocessed to obtain a preprocessed signal. The preprocessed signal includes several sampling points, each of which has a corresponding amplitude and a corresponding sampling point number.

[0007] After performing local extremum detection on the preprocessed signal according to the sampling point numbering order, a set of maxima points and a set of minima points are obtained; the set of maxima points includes several local maxima sampling points; the set of minima points includes several local minima sampling points.

[0008] Based on the set of maximum points and the set of minimum points, and the first preset model, one of the set of maximum points and the set of minimum points is selected as the selected extreme point set;

[0009] The selected extreme point set is filtered according to the second preset model to obtain the filtered extreme point set;

[0010] The fundamental frequency F0 and time-domain smoothing are performed on the filtered extreme point set.

[0011] Preferably, before preprocessing the received frame of sound signal, the method further includes: performing preliminary screening on the sound signal to remove fundamental frequency detection errors.

[0012] Preferably, the preprocessing of the received frame of audio signal includes: downsampling, low-pass filtering, and high-pass filtering.

[0013] Preferably, the downsampling sampling frequency is not less than 1600Hz.

[0014] Preferably, the first preset model is:

[0015] G1 = val(val > median(val)) (1)

[0016] G2=val(val≤median(val)) (2)

[0017] F=|median(G1)-median(G2)| (3)

[0018] Where val represents the maximum sampling point in the set of maximum points or the minimum sampling point in the set of minimum points, median() represents the median value, and median(val) represents the median of the amplitude of all maximum sampling points or the median of the amplitude of all minimum sampling points, that is, the extreme median of the maximum or the extreme median of the minimum.

[0019] G1 represents the first set of maximum / minimum sample points with amplitudes greater than median(val); G2 represents the second set of maximum / minimum sample points with amplitudes less than median(val); F represents the absolute value of the difference between the median amplitude of the first set of maximum / minimum sample points and the median amplitude of the second set of maximum / minimum sample points.

[0020] When the absolute value of the difference between the maximum point set is greater than the absolute value of the difference between the minimum point set, the maximum point set is selected as the selected extreme point set; otherwise, the minimum point set is selected as the selected extreme point set.

[0021] Preferably, the second preset model is:

[0022]

[0023] Where k represents a selected sampling point in the selected extreme point set, val(k) represents the amplitude corresponding to the selected sampling point, val(k-1) represents the amplitude corresponding to the sampling point to the left of the selected sampling point in the selected extreme point set, and val(k+1) represents the amplitude corresponding to the sampling point to the right of the selected sampling point in the selected extreme point set.

[0024] If val(k) / val(k-1) is less than 0.5 or val(k) / val(k+1) is less than 0.5, then the selected sampling point is excluded; otherwise, it is retained and the following steps are performed.

[0025] When the selected sampling point is a local minimum sampling point, the following conditions are met:

[0026] val(k)>0.7val(k-1)AND val(k)>0.7val(k+1) (5);

[0027] Or, when the selected sampling point is a local maximum sampling point, the following condition is met:

[0028] val(k)<0.7val(k-1)AND val(k)<0.7val(k+1) (6);

[0029] The selected sampling points are excluded; otherwise, they are retained and the following steps are performed:

[0030] When the selected sampling point is a local minimum sampling point, the following conditions are met:

[0031] TH = 0.5 min (val) (7)

[0032] val(k)>0.8TH (8)

[0033] Or, when the selected sampling point is a local maximum sampling point, the following condition is met:

[0034] TH = 0.5max(val) (9)

[0035] val(k)<0.8TH (10)

[0036] Where TH is the set threshold, min() represents finding the minimum value, and max() represents finding the maximum value;

[0037] The selected sampling points are excluded; otherwise, they are retained.

[0038] All the selected sampling points that are retained are aggregated into the filtered extreme point set.

[0039] Preferably, the specific steps for calculating the fundamental frequency F0 and performing time-domain smoothing are as follows:

[0040] f0=Fs / median(diff(pos)) (11)

[0041] Where Fs is the sampling rate after downsampling, pos is the index of the sampling point position of the filtered extreme point set in the downsampled signal, and diff(pos) means performing a first-order difference on pos to obtain a first-order difference signal.

[0042] Then, a time-domain recursive approach is used to smooth the result of f0, as shown in the following formula.

[0043] F0(i)=alphaF0(i-1)+f0(1-alpha) (12)

[0044] Where F0(i-1) is the F0 estimation result of the previous frame, alpha is the smoothing factor, and to prevent outlier interference, different levels of updates are set according to the difference between the f0 value and F0(i-1).

[0045] A real-time speech fundamental frequency contour extraction system employs the aforementioned method. The system includes a preprocessing module and a fundamental frequency detection module.

[0046] The preprocessing module is used to preprocess the acoustic signal to obtain the preprocessed signal;

[0047] The baseband detection module is used to detect the baseband of the preprocessed signal to obtain the baseband.

[0048] Preferably, the system further includes a signal screening module connected to the preprocessing module. Before the preprocessing module preprocesses the received acoustic signal, the signal screening module performs preliminary screening on the received acoustic signal to remove fundamental frequency detection errors.

[0049] Preferably, the baseband detection module includes a local extremum detection unit, an extremum determination unit, and an extremum filtering unit connected in sequence;

[0050] The local extremum detection unit is used to perform local extremum detection on the preprocessed unit and obtain the set of maximum and minimum points;

[0051] The extreme value determination unit is used to filter the maximum value set and the minimum value set according to the first preset model, and select one of them as the selected extreme value set;

[0052] The extreme value screening unit is used to screen the selected extreme value point set according to the second preset model and obtain the screened extreme value point set.

[0053] The beneficial effects of this invention are as follows:

[0054] 1. The present invention provides a real-time speech fundamental frequency contour extraction method, which realizes speech fundamental frequency estimation by directly detecting and filtering the local extreme value positions of the time-domain waveform of the preprocessed sound signal. Under the premise of not significantly affecting the accuracy of fundamental frequency calculation, it greatly reduces the computational overhead of real-time fundamental frequency detection and is more conducive to running on terminal devices with low computing power and low power consumption requirements.

[0055] 2. In order to prevent fundamental frequency detection from being performed in environments with weak signals or high noise levels, the method of real-time speech fundamental frequency contour extraction of the present invention first performs preliminary screening of the sound signal before preprocessing, so as to avoid possible fundamental frequency detection errors that may cause incorrect control of the channel envelope amplitude, thereby potentially causing adverse effects on the hearing effect of the implantee.

[0056] 3. In the real-time speech fundamental frequency contour extraction method of the present invention, in order to meet the real-time computing requirements and have a certain number of fundamental frequency cycles, the signal is first downsampled from 16kHz to 1600Hz. Thus, the length of one frame is 48 sampling points, which effectively meets the real-time computing requirements.

[0057] 4. The real-time speech fundamental frequency contour extraction method of the present invention, by setting a unique fundamental frequency detection method, namely by extreme value position detection, using the position information of the maximum value point set or the minimum value point set, and extreme value position filtering, effectively eliminates the computing power requirements of conventional F0 detection methods, greatly reduces the computing resource consumption of the device, and is more suitable for real-time operation in devices with low computing power and low power consumption requirements such as cochlear implants.

[0058] 5. The real-time speech fundamental frequency contour extraction system of the present invention adopts the above-mentioned real-time speech fundamental frequency contour extraction method, which effectively performs initial screening, local extremum detection, extremum determination and extremum screening of sound signals. Compared with the classic AMDF method, it greatly reduces the computational overhead of real-time F0 detection without significantly affecting the accuracy of F0 calculation, and is more conducive to running on terminal devices with low computing power and low power consumption requirements. Attached Figure Description

[0059] Figure 1 This is a schematic diagram of the overall process of a real-time speech fundamental frequency contour extraction method according to a specific embodiment of the present invention;

[0060] Figure 2 This is a waveform diagram of a real-time speech fundamental frequency contour extraction method according to a specific embodiment of the present invention;

[0061] Figure 3 This is a schematic diagram comparing the fundamental frequency recognition results of a real-time speech fundamental frequency contour extraction method according to a specific embodiment of the present invention;

[0062] Figure 4 This is a schematic diagram comparing the fundamental frequency recognition results of a real-time speech fundamental frequency contour extraction method according to a specific embodiment of the present invention. Detailed Implementation

[0063] The preferred embodiments of the present invention will now be described in detail with reference to the accompanying drawings.

[0064] refer to Figure 1 A real-time speech fundamental frequency contour extraction method includes the following steps:

[0065] S101, Determine whether the input signal meets the fundamental frequency detection conditions: Before preprocessing the received frame of sound signal, the sound signal is initially screened to remove fundamental frequency detection errors; specifically as follows: Input signal, determine whether the input sound signal meets the fundamental frequency detection conditions, if yes, proceed to the next step S102; if no, do not proceed.

[0066] S102, Signal preprocessing: The received frame of sound signal is preprocessed to obtain a preprocessed signal. The preprocessed signal includes several sampling points, each of which has a corresponding amplitude and a corresponding sampling point number.

[0067] S103, baseband F0 detection, the specific steps are as follows:

[0068] S1031, Local Extremum Location Detection: Local extremum detection is performed on the preprocessed signal according to the sampling point numbering order to obtain a set of maxima and a set of minima; the set of maxima includes several local maxima sampling points; the set of minima includes several local minima sampling points; wherein, both local maxima sampling points and local minima sampling points belong to the definition of local extremum points of the function.

[0069] S1032, Determine whether to use the location of the maximum or minimum point set: Based on the maximum and minimum point sets and the first preset model, select one of the maximum and minimum point sets as the selected extreme point set;

[0070] S1033, Extreme value location filtering: The selected extreme value point set is filtered according to the second preset model, and the filtered extreme value point set is obtained;

[0071] S1034 performs fundamental frequency F0 calculation and time-domain smoothing on the filtered extreme point set.

[0072] Specifically, in step S101, to prevent fundamental frequency detection from being performed in environments with weak signals or high noise levels, and to avoid potential fundamental frequency detection errors causing incorrect modulation of the channel envelope amplitude, which could potentially have adverse effects on the implant recipient's hearing, the algorithm first determines whether to proceed with the subsequent fundamental frequency extraction step using some simple energy-based judgment rules, as follows:

[0073] a) Perform simple background noise level detection, such as by detecting the minimum energy value within a certain time period. If the background noise level exceeds the set threshold (e.g., 60dB SPL), then do not proceed with the subsequent steps; where SPLl (sound pressure level) stands for sound pressure level.

[0074] b) If the background noise level is below the threshold, then further determine whether the energy of the current frame exceeds the background noise level to a certain extent, such as by detecting the real-time signal-to-noise ratio. If the current signal-to-noise ratio is below the set threshold (e.g., 10dB), then do not proceed with the subsequent steps.

[0075] c) Determine whether the energy of the current frame exceeds a preset absolute threshold. If it is lower than the set threshold, skip subsequent F0 detection and other steps.

[0076] In step S102, the preprocessing of the received frame of audio signal includes downsampling, low-pass filtering, and high-pass filtering. To meet real-time computation requirements and ensure a sufficient number of pitch cycles, considering both computational complexity and low error, the downsampling frequency is generally not less than 1600Hz, preferably 1600Hz. In a specific embodiment, a duration of 30m is selected as the signal length of one frame. First, the signal is downsampled from a 16kHz sampling rate to 1600Hz, resulting in a frame length of 48 sampling points. Then, a 3rd-order Butterworth low-pass filter (cutoff frequency 500Hz) and a 5th-order Butterworth high-pass filter (cutoff frequency 50Hz) are performed.

[0077] In step S103, baseband F0 detection is underway.

[0078] See Figure 2 The detailed steps of step S1031 are as follows:

[0079] Based on observations of the time-domain waveforms of audio data, some parts are suitable for gene cycle detection using maximum location information, while others are more suitable using minimum location information. Therefore, in one specific embodiment, for the preprocessed signal waveform data of a frame of audio signal, the positions of its local maxima and minima are simultaneously detected. The sampling points from the 2nd to the 47th are iteratively traversed. If the amplitude corresponding to the current sampling point k is simultaneously higher than its two adjacent sampling points (k-1 and k+1), it is marked as a local maximum sampling point, and the corresponding maximum amplitude is recorded. Conversely, if the amplitude corresponding to the current sampling point k is simultaneously lower than its two adjacent sampling points (k-1 and k+1), it is marked as a local minimum sampling point, and the corresponding minimum amplitude is recorded. It is worth noting that the amplitude defined here and below, i.e., the amplitude in the implementation method, can also be a negative value less than zero.

[0080] The detailed steps of step S1032 are as follows:

[0081] Based on the first preset model, it is determined whether to use the maximum or minimum point set for gene cycle calculation. The specific method is as follows: For the maximum point set, firstly, an amplitude threshold is set as the median value of the maximum values. Then, the points are divided into two groups based on whether the values ​​exceed this amplitude threshold. The median values ​​of these two maximum point sets are calculated separately, and then the absolute value of the difference between these two median values ​​is calculated. The extraction of the specific values ​​of the minimum value vector is performed in the same way. The operation of the first preset model is executed on both the maximum and minimum point sets. The first preset model is...

[0082] G1 = val(val > median(val)) (1)

[0083] G2=val(val≤median(val)) (2)

[0084] F=|median(G1)-median(G2)| (3)

[0085] Where val represents the maximum sampling point in the set of maximum points or the minimum sampling point in the set of minimum points, median() represents the median value, and median(val) represents the median of the amplitude of all maximum sampling points or the median of the amplitude of all minimum sampling points, that is, the extreme median of the maximum or the extreme median of the minimum.

[0086] G1 represents the first set of maximum / minimum sample points with amplitudes greater than median(val); G2 represents the second set of maximum / minimum sample points with amplitudes less than median(val); F represents the absolute value of the difference between the median amplitude of the first set of maximum / minimum sample points and the median amplitude of the second set of maximum / minimum sample points.

[0087] Specifically, when val represents a maximum sampling point, G1 represents the first batch of maximum sampling points with an amplitude greater than median(val); G2 represents the second batch of maximum sampling points with an amplitude less than median(val); and F represents the absolute value of the difference between the median amplitude of the first batch of maximum sampling points and the median amplitude of the second batch of maximum sampling points. When val represents a minimum sampling point, G1 represents the first batch of minimum sampling points with an amplitude greater than median(val); G2 represents the second batch of minimum sampling points with an amplitude less than median(val); and F represents the absolute value of the difference between the median amplitude of the first batch of minimum sampling points and the median amplitude of the second batch of minimum sampling points.

[0088] When the absolute value of the difference between the maximum points is greater than the absolute value of the difference between the minimum points, the maximum points are selected as the selected extreme points; otherwise, the minimum points are selected as the selected extreme points.

[0089] In step S1033, the detailed steps are as follows:

[0090] To prevent frequency doubling issues, the detected extreme value positions are screened, and possible half-cycle positions are removed according to set rules. In this specific embodiment, the judgment criterion, i.e., the second preset model, sets three calculation and screening steps, which are performed sequentially, as follows:

[0091] First, calculate the ratio of the current extreme value to its left and right adjacent extreme values. If either ratio is lower than a set threshold of 0.5, then this extreme value is excluded, as follows:

[0092]

[0093] Where k represents a selected sampling point in the selected extreme point set, val(k) represents the amplitude corresponding to the selected sampling point, val(k-1) represents the amplitude corresponding to the sampling point to the left of the selected sampling point in the selected extreme point set, that is, the amplitude corresponding to the sampling point one position before the selected sampling point in the selected extreme point set; val(k+1) represents the amplitude corresponding to the sampling point to the right of the selected sampling point in the selected extreme point set, that is, the amplitude corresponding to the sampling point one position after the selected sampling point in the selected extreme point set;

[0094] If val(k) / val(k-1) is less than 0.5 or val(k) / val(k+1) is less than 0.5, then the selected sampling point is excluded; otherwise, it is retained and the following steps are performed:

[0095] When a sampling point is selected as a local minimum sampling point, the current minimum value must be higher than 70% of the two adjacent minimum values, as follows:

[0096] val(k)>0.7val(k-1)AND val(k)>0.7val(k+1) (5);

[0097] Alternatively, when the selected sampling point is a local maximum sampling point, if the current maximum value is simultaneously lower than 70% of the two adjacent maximum values, the specific conditions are as follows:

[0098] val(k)<0.7val(k-1)AND val(k)<0.7val(k+1) (6);

[0099] If selected sampling points are excluded, otherwise they are retained and the following steps are performed:

[0100] When the selected sampling point is a local minimum sampling point, if the current minimum value is higher than a set threshold of 80%, the threshold is adaptively set to 50% of the minimum value among all detected minimum values, as follows:

[0101] TH = 0.5 min (val) (7)

[0102] val(k)>0.8TH (8)

[0103] Alternatively, when the selected sampling point is a local maximum sampling point, if the current maximum value is lower than 80% of a set threshold, the threshold is adaptively set to 50% of the maximum value among all detected maxima, as follows:

[0104] TH = 0.5max(val) (9)

[0105] val(k)<0.8TH (10)

[0106] Where TH is the set threshold, min() represents finding the minimum value, and max() represents finding the maximum value;

[0107] The selected sampling points are excluded; otherwise, they are retained.

[0108] All the selected retained sampling points are aggregated into the filtered extreme point set.

[0109] See Figure 2 This example demonstrates a local extremum detection and filtering process. Specifically, a) local maxima (green dots) and minima (black dots) are detected in a preprocessed frame of waveform data, with the upper and lower black horizontal lines representing the median values ​​of the local maxima and minima, respectively; b) based on a first preset model (in this example, the absolute value of the difference between the maxima set is 0.0443, the absolute value of the difference between the minima set is 0.4521, and the absolute value of the difference between the maxima set is less than the absolute value of the difference between the minima set), the minima set is selected as the chosen extremum set.

[0110] Furthermore, after determining the set of minimum points as the selected set of extreme points, the second preset model is used for further screening: when the selected sampling point in the selected set of extreme points is a local minimum sampling point k, val(k) represents the amplitude corresponding to the selected sampling point, val(k-1) represents the amplitude corresponding to the local minimum sampling point k-1 that is the position before (left) of the nearest local minimum sampling point k in the selected set of extreme points, and val(k+1) represents the amplitude corresponding to the local minimum sampling point that is the position after (right) of the nearest local minimum sampling point k in the selected set of extreme points.

[0111] After calculation and filtering according to the second preset model mentioned above, all the selected sampling points that are retained are gathered into a set of extreme points after filtering. Figure 2 In the diagram, the black line represents the median of all local minimum amplitudes. The red dots indicate the locations of local minimum sampling points retained after extreme value screening according to the second preset model (i.e., the set of extreme value points after screening), while the black dots indicate the locations of discarded local minimum sampling points.

[0112] In step S1034, the detailed steps for calculating the fundamental frequency F0 and performing time-domain smoothing on the filtered extreme point set are as follows:

[0113] f0 = Fs / median(diff(pos))

[0114] Where Fs is the sampling rate after downsampling, pos is the index of the sampling point position of the filtered extreme point set in the downsampled signal, and diff(pos) means performing a first-order difference on pos to obtain a first-order difference signal.

[0115] Then, a time-domain recursive approach is used to smooth the result of f0, as shown in the following formula.

[0116] F0(i)=alphaF0(i-1)+f0(1-alpha) (9)

[0117] Where F0(i-1) is the F0 estimation result of the previous frame, and alpha is a smoothing factor. To prevent interference from outliers, different levels of updates are set according to the difference between the f0 value and f0(i-1). If the absolute value of the difference is higher than the set threshold, a slower update is performed and the alpha value is set to be larger; otherwise, a faster update is performed.

[0118] This invention also provides a real-time speech fundamental frequency contour extraction system. Employing the aforementioned real-time speech fundamental frequency contour extraction method, the system includes a preprocessing module and a fundamental frequency detection module connected sequentially.

[0119] The preprocessing module is used to preprocess the received acoustic signal to obtain the preprocessed signal;

[0120] The baseband detection module is used to detect the baseband of the preprocessed signal to obtain the baseband.

[0121] Furthermore, it also includes a signal screening module connected to the preprocessing module. Before the preprocessing module preprocesses the received acoustic signal, the signal screening module performs preliminary screening on the received acoustic signal to remove fundamental frequency detection errors.

[0122] Furthermore, the baseband detection module includes a local extremum detection unit, an extremum determination unit, and an extremum filtering unit connected in sequence;

[0123] The local extremum detection unit is used to detect local extrema in the preprocessed units and obtain the set of maximum and minimum points;

[0124] The extreme value determination unit is used to filter the set of maximum and minimum points according to the first preset model, and select one of them as the selected extreme value set;

[0125] The extreme value filtering unit is used to filter the selected extreme value point set according to the second preset model and obtain the filtered extreme value point set.

[0126] Furthermore, this application uses experimental results to illustrate its effectiveness. LEDD (Local Extrema Detection and Decision) is the local extrema detection and decision algorithm of this application.

[0127] 1. Estimation of computational complexity for F0 detection based on LEDD:

[0128] The F0 detection module in the Nurotron cochlear implant tone perception enhancement coding C-tone algorithm reduces the sampling rate from 16kHz to 8000Hz by downsampling, while using a 32ms time window (corresponding to 256 sampling points) for fundamental frequency detection. It also reduces the computational load by a factor of 4 by sampling the AMDF. Even so, the formula requires (32+31+32)*128=12160 arithmetic operations to calculate the dd value, including addition, subtraction and absolute value calculation.

[0129]

[0130] The local extremum-based method proposed in this invention, in a specific embodiment, reduces the sampling rate to 1600Hz, uses a 30ms time window, corresponding to 48 sampling points. Extremum detection requires 46*2=92 comparisons. The subsequent extremum filtering process, following conventional methods, detects no more than 16 extremum points, requiring a maximum of 16*12=192 arithmetic operations, including comparisons, multiplication, division, and absolute value calculations. Thus, the entire fundamental frequency determination process requires approximately only 300 arithmetic operations, reducing the computational load to 1 / 40th of the sampling AMDF algorithm used in C-tone, significantly reducing the computational complexity.

[0131] 2. Evaluation of F0 detection accuracy based on LEDD:

[0132] Theoretically, with a significant reduction in computational load, the accuracy of F0 detection will inevitably decrease. To assess the extent of the impairment in F0 detection accuracy of the LEDD algorithm, Praat speech analysis software was used to automatically detect the F0 of monosyllabic speech. Then, obvious detection errors or undetectable phenomena were manually corrected through visual observation, serving as the standard value for the F0 result. This result was compared with the results of the LEDD and C-tone algorithms. The Mean Absolute Error Score (MAES) was calculated to evaluate the degree to which the two algorithms deviated from the standard value.

[0133]

[0134] To align different F0 detection results in time, linear interpolation is performed on the F0 results automatically detected by each algorithm. In Formula 10, the result is the linear interpolation of the fundamental frequency result automatically detected by the original LEDD algorithm or C-tone algorithm, and the result is the linear interpolation of the standard value automatically detected by Praat software and manually corrected. The specific method is as follows: a) Obtain the start and end times of the analysis window, which depends on the three F0 detection results. Assuming that the start times of the valid F0 values ​​(non-zero values, non-NaN values) detected by the Praat, C-tone, and LEDD algorithms are ts1, ts2, and ts3, and the end times are te1, te2, and te3, respectively, then the start time TS of the analysis window is max(ts1, ts2, ts3), and the end time TE is min(te1, te2, te3). b) Obtain a unified F0 calculation time vector T by performing linear interpolation on the time range of TS~TE (step size set to 15ms). The N value in Formula 10 is the length value of T; c) Calculate the F0 results of each algorithm for T by linear interpolation.

[0135] Two monosyllabic corpora were used to evaluate accuracy. One was a female monosyllabic corpus from the MSP software, containing 332, 257, 320, and 351 monosyllabic words (a total of 1260) in tones 1 to 4, recorded by the same female announcer. To evaluate the algorithm's results in detecting the fundamental frequency of male voices, another monosyllabic audio library was used, derived from the Angel Speech Training software. This library contained 64 monosyllabic words, with syllables consisting of the fixed initial consonant 'b' and four simple vowels 'a, o, i, u', each with four tones, and broadcast by two men and two women respectively.

[0136] See Figure 3 This shows a comparison example of the F0 recognition results of three algorithms (Praat, Norcomc C-tone, and LEDD) in various tones of MSP monosyllabic sounds. Figure 3 The statistical distribution of the detection errors MAES of the C-tone and LEDD algorithms F0 (a~d) and (a~d) Figure 3 It can be seen that the recognition error distribution of LEDD and C-tone algorithms is relatively concentrated in tone 1 and tone 2, and the error scores are relatively small, indicating that the F0 detection results are relatively accurate. The next most accurate result is for tone 3, while the MAES result distribution for tone 4 is the most diffuse and the error value is relatively high compared to other tones, especially for the C-tone algorithm. The statistical results of MAES calculation for the two algorithms under each tone are shown in Table 1.

[0137] Table 1. Detection error of the two algorithms for each tone F0 of monosyllabic words in MSP. MASE statistical results.

[0138]

[0139]

[0140] Because the data in each group did not meet the requirement of normality (Kolmogorov-Smirnov test, p < 0.001), the non-parametric Scheirer-Ray-Hare test was used. The results showed that the MAES results of the F0 detection error of different algorithms were significantly different (H = 421.6, p < 0.001), and there was also a significant interaction effect between the algorithm and tone (H = 199.9, p < 0.001). To investigate this interaction effect, we used the Mann-Whitney U test for post-hoc pairwise comparisons at each tone level and used the Bonferroni method to perform multiple comparison correction for the p-values. The results showed that C-tone and LEDD performed significantly differently under the conditions of tone 1 (p_adj < 0.001), tone 2 (p_adj < 0.001), and tone 3 (p_adj < 0.001). However, no significant difference was found under the condition of tone 4 (p_adj > 0.05). Based on all tone data, there is a significant difference in the F0 detection performance of the two algorithms (Mann-Whitney U test, z = -20.534, p < 0.001). The F0 recognition error of C-tone is significantly lower than that of LEDD. The median recognition error score of C-tone is 0.017 with an interquartile range of 0.026, while the median recognition error of LEDD is 0.046 with an interquartile range of 0.038. Overall, the LEDD algorithm improves the F0 detection error of MSP monosyllabic speech by about 3% compared with the C-tone algorithm.

[0141] Angel Language Training Monosyllabic Word Recognition Results:

[0142] The C-tone and LEDD algorithms were used to perform F0 recognition of male and female monosyllabic words, respectively, and the MAES values ​​of the F0 recognition results were calculated. The specific statistical values ​​are shown in Table 2, including the mean, variance, median and interquartile range.

[0143] Table 2. MAES statistical results of the detection errors of the two algorithms for each tone F0 of monosyllabic words in male and female voices in Angel Language Training.

[0144]

[0145] Figure 4 (a, b) show two examples of three algorithms detecting male monosyllabic pitch 2 and pitch 3F0. Figure 4The violin plots (c, d) show the data distribution of MAES for F0 detection by the two algorithms. It can be seen that the overall detection error distribution of both algorithms is more concentrated and the values ​​are relatively lower for female voices, suggesting that the two algorithms are more accurate in detecting F0 for female voices than male voices. Furthermore, in female voice monosyllabic F0 recognition, both algorithms have lower accuracy in tone 4 compared to the other three tones, which is consistent with the previous MSP conclusion. In male voice monosyllabic F0 recognition, the C-tone algorithm exhibits larger recognition errors in tones 2 and 3 compared to the other two tones, while the LEDD algorithm does not exhibit this phenomenon. The Wilcoxon rank-sum test was used to analyze the difference in MAES for monosyllabic F0 detection results of the two algorithms for each tone in both male and female voices. The original p-values ​​are shown in Table 3.

[0146] Table 3. Significant difference in MAES values ​​of F0 recognition error between the two algorithms for different genders and voice tones.

[0147]

[0148] Note: The bold black text indicates that the results still show significant differences after the Benjamini-Hochberg correction procedure.

[0149] Referring to Table 3, when the Benjamini-Hochberg correction method was used to control the error rate under multiple comparisons to below 5%, the results showed significant differences in the MAES results of the two algorithms for female monosyllable recognition across all tones. Specifically, for F0 detection of tones 1 to 3, the detection error of the C-tone algorithm was significantly lower than that of the LEDD algorithm, while the opposite was true for tone 4. For male monosyllable tone F0 detection errors, there was no significant difference between the two algorithms for tones 1 and 4, while for tones 2 and 3, the F0 detection error of the LEDD algorithm was significantly lower than that of the C-tone algorithm. Considering all tones of both male and female voices, there was no significant difference in the F0 detection error between the C-tone and LEDD algorithms (Mann-Whitney U test, z = -0.998, p > 0.5). Based on the overall statistical analysis, the median MAES of the C-tone algorithm is 0.033 and the interquartile range is 0.458, while the median MAES of the LEDD algorithm is 0.062 and the interquartile range is 0.142. This means that the F0 detection error of LEDD is about 0.03 higher than that of C-tone, but the detection results are more stable.

[0150] In summary, the real-time speech fundamental frequency contour extraction method and system of the present invention achieves speech fundamental frequency estimation by directly detecting and filtering the local extreme value positions of the preprocessed audio signal time-domain waveform. This greatly reduces the computational overhead of real-time fundamental frequency detection without significantly affecting the accuracy of fundamental frequency calculation, and is more conducive to running on terminal devices with low computing power and low power consumption requirements.

[0151] Finally, it should be noted that the above preferred embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail through the above preferred embodiments, those skilled in the art should understand that various changes can be made to it in form and detail without departing from the scope defined by the claims of the present invention.

Claims

1. A real-time speech fundamental frequency contour extraction method, characterized in that, Includes the following steps: The received frame of sound signal is preprocessed to obtain a preprocessed signal. The preprocessed signal includes several sampling points, each of which has a corresponding amplitude and a corresponding sampling point number. After performing local extremum detection on the preprocessed signal according to the sampling point numbering order, a set of maxima points and a set of minima points are obtained; the set of maxima points includes several local maxima sampling points; the set of minima points includes several local minima sampling points. Based on the set of maxima and the set of minima, and a first preset model, one of the set of maxima and the set of minima is selected as the selected extreme point set: the operation of the first preset model is performed on the set of maxima and the set of minima respectively, wherein the first preset model is: ； in, val This represents a maximum sampling point in a set of maximum points or a minimum sampling point in a set of minimum points. median () represents finding the median value. median ( val ) represents the median of the amplitude of all maximum sampling points or the median of the amplitude of all minimum sampling points, that is, the extreme median of the maximum or the extreme median of the minimum. G 1 indicates that the amplitude is greater than median ( val The first set of maximum / minimum sampling points; G 2 indicates that the amplitude is less than median ( val The second set of maximum / minimum sampling points; F represents the absolute value of the difference between the median amplitude of the first set of maximum / minimum sampling points and the median amplitude of the second set of maximum / minimum sampling points; The selected extreme point set is filtered according to the second preset model to obtain the filtered extreme point set; The second preset model is: ； in, k This represents a selected sampling point within the selected extreme point set. val ( k ) represents the amplitude corresponding to the selected sampling point. val ( k -1) represents the amplitude corresponding to the sampling point closest to the left of the selected sampling point in the selected extreme point set. val ( k +1) represents the amplitude corresponding to the sampling point to the right of the nearest selected sampling point in the selected extreme point set; The fundamental frequency F0 and time-domain smoothing are performed on the filtered extreme point set.

2. The real-time speech fundamental frequency contour extraction method according to claim 1, characterized in that, Before preprocessing the received frame of sound signal, the method further includes: performing preliminary screening on the sound signal to remove fundamental frequency detection errors.

3. The real-time speech fundamental frequency contour extraction method according to claim 1, characterized in that, The preprocessing of the received frame of audio signal includes downsampling, low-pass filtering, and high-pass filtering.

4. The real-time speech fundamental frequency contour extraction method according to claim 3, characterized in that, The downsampling sampling frequency is not less than 1600Hz.

5. The real-time speech fundamental frequency contour extraction method according to claim 1, characterized in that, When the absolute value of the difference between the maximum point set is greater than the absolute value of the difference between the minimum point set, the maximum point set is selected as the selected extreme point set; otherwise, the minimum point set is selected as the selected extreme point set.

6. The real-time speech fundamental frequency contour extraction method according to claim 1, characterized in that, like val ( k ) / val ( k -1) Less than 0.5 or val ( k ) / val ( k If +1) is less than 0.5, then the selected sampling point is excluded; Otherwise, retain the following steps; When the selected sampling point is a local minimum sampling point, the following conditions are met: ； Or, when the selected sampling point is a local maximum sampling point, the following condition is met: ； The selected sampling points are excluded; otherwise, they are retained and the following steps are performed: When the selected sampling point is a local minimum sampling point, the following conditions are met: ； Or, when the selected sampling point is a local maximum sampling point, the following condition is met: ； in, TH To set a threshold, min() represents finding the minimum value, and max() represents finding the maximum value; The selected sampling points are excluded; otherwise, they are retained. All the selected sampling points that are retained are aggregated into the filtered extreme point set.

7. The real-time speech fundamental frequency contour extraction method according to claim 1, characterized in that, The specific steps for calculating the fundamental frequency F0 and performing time-domain smoothing are as follows: ； in Fs To reduce the sampling rate after downsampling, pos This refers to the index of the sampling point positions in the downsampled signal for the filtered set of extreme points. diff ( pos ) indicates to pos Perform first-order difference to obtain the first-order difference signal; Then, a time-domain recursive approach was used to... f The result is smoothed to a certain extent, as shown in the following formula. ； in, F 0( i -1) represents the F0 estimation result for the previous frame. alpha As a smoothing factor, it is set according to the following to prevent interference from outliers. f 0 value and F 0( i The difference between -1) is updated to different degrees.

8. A real-time speech fundamental frequency contour extraction system, characterized in that, The system employs the method described in any one of claims 1-7, wherein the system comprises a preprocessing module and a baseband detection module, wherein, The preprocessing module is used to preprocess the acoustic signal to obtain the preprocessed signal; The baseband detection module is used to detect the baseband of the preprocessed signal to obtain the baseband.

9. The real-time speech fundamental frequency contour extraction system according to claim 8, characterized in that, It also includes a signal screening module connected to the preprocessing module. Before the preprocessing module preprocesses the received acoustic signal, the signal screening module performs preliminary screening on the received acoustic signal to remove fundamental frequency detection errors.

10. The real-time speech fundamental frequency contour extraction system according to claim 8, characterized in that, The base frequency detection module includes a local extreme value detection unit, an extreme value determination unit, and an extreme value filtering unit connected in sequence. The local extremum detection unit is used to perform local extremum detection on the preprocessed unit and obtain the set of maximum and minimum points; The extreme value determination unit is used to filter the maximum value set and the minimum value set according to the first preset model, and select one of them as the selected extreme value set; The extreme value screening unit is used to screen the selected extreme value point set according to the second preset model and obtain the screened extreme value point set.

Images