Method and system for dynamic feature extraction of audio signals

Abstract

The present application relates to a method and system for dynamic feature extraction of audio signals. A method of processing an audio signal is disclosed, comprising: pre-processing an input audio sequence signal over a time period, including dividing the audio sequence signal over the time period into a plurality of audio signal frames; classifying each of the plurality of audio signal frames into at least a significant frame or a non-significant frame based on at least a salience value of one or more audio signal frames; performing feature extraction on a first proportion of audio signals in each significant frame using a high-level feature extraction process; performing feature extraction on a second proportion of audio signals in each non-significant frame using a low-level feature extraction process, wherein the second proportion is less than the first proportion and the second proportion is greater than zero; and matching and integrating at least features extracted from the significant frames and features extracted from the non-significant frames to generate a feature vector for the audio sequence signal over the time period.

Description

Technical Field

[0001] This disclosure generally relates to the field of speech recognition, and more specifically, to a method and system for dynamic feature extraction of audio signals. Background Technology

[0002] A typical automatic speech recognition system consists of two main parts: feature extraction and a recognition engine. Feature extraction is the first step in speech recognition, and its main purpose is to extract the recognizable components from the audio signal and remove other parts such as background noise and emotional states, so that the recognition engine can be trained and used for speech recognition.

[0003] The market for Artificial Intelligence of Things (AIoT) applications is growing steadily and is projected to reach 16.2 billion by 2024. Edge AI is becoming a key focus for future AIoT development, with high efficiency, low power consumption, and strong cloud integration becoming the prevailing trends. Low-hardware-resource binary neural network (BNN) architectures are crucial for IoT applications. For AIoT applications such as voice-activity-detection (VAD) and keyword-spotting (KWS), low-complexity neural networks are already being used in the recognition engines. Therefore, the feature extraction part, comparatively, consumes a significant portion of the overall system's computational resources.

[0004] Since the computation of the recognition engine in an automatic speech recognition system has been simplified by using low-complexity neural network processing, this disclosure aims to reduce the computational complexity of the feature extraction part in order to further reduce the computational resource and power consumption requirements of the automatic speech recognition system.

[0005] This disclosure focuses on time-frequency feature extraction, including short-time Fourier transform (including Gabor transform), discrete cosine transform (DCT), wavelet transform, bilinear time-frequency distribution function (Wigner distribution function, or WDF), modified Wigner distribution function, Gabor-Wigner distribution function, etc. (see Gabor-Wigner transform), Hilbert-Huang transform, Walsh-Hadamard transform (WHT), modulated complex overlap transform (MCLT), and other methods.

[0006] Mel frequency cepstral coefficients (MFCCs) are a feature extraction method widely used in automatic speech recognition to date. Figure 10The functional modules of a conventional MFCC are shown. The conventional MFCC feature extraction process includes a preprocessing module 1001, which preprocesses the input speech signal, including pre-emphasis, framing, and windowing; an FFT module 1002, which obtains the corresponding spectrum for each short-time analysis window using a Fast Fourier Transform (FFT); a Mel filter 1003, which passes the obtained spectrum through a Mel filter bank to obtain the Mel spectrum; a Log module 1004, which takes the logarithm of the Mel spectrum; and a Discrete Cosine Transform (DCT) module 1005, which performs an inverse DCT to perform cepstral analysis, thereby obtaining the Mel frequency cepstral coefficients (MFCC). These Mel frequency cepstral coefficients (MFCC) are the feature vectors of the speech frame. The input speech signal can be described by a series of cepstral vectors. Each vector is the MFCC feature vector for each frame. After obtaining the feature vectors, a recognition engine can be used to train these feature vectors for speech recognition.

[0007] like Figure 10 The MFCC feature extraction process shown uses FFT, Mel filter, and DCT modules, which require a significant amount of computation. For speech recognition, the computational complexity of MFCC is almost comparable to that of Convolutional Neural Networks (CNNs).

[0008] For relatively simple speech recognition applications, much research has been conducted to simplify, for example, the MFCC feature extraction process to reduce computational complexity. For instance, by simplifying... Figure 10 The MFCC speech feature extraction shown can be simplified by removing certain functional modules or one or more functional modules from the feature extraction path to reduce computational complexity.

[0009] CN102332262B discloses a song intelligent recognition method based on audio features. It discloses a simplified MFCC feature extraction method for all frames of the input speech signal to reduce computational complexity. Specifically, CN102332262B samples and frames the audio data stream. For each frame's sampling points, a Fast Fourier Transform (FFT) is performed, resulting in N frequency domain amplitude values. These are then divided into y+1 equal parts according to their respective frequency bands. The sum of the absolute values ​​S of each frequency domain value within each part of the frequency band is calculated, and y feature values ​​are extracted for each audio signal frame. Compared to the conventional MFCC feature vector extraction process, this patent CN102332262B omits windowing, Mel filters, and DCT modules in the preprocessing module, and instead uses only the sum (absolute value) of the frequency domain amplitude values ​​after FFT of each frame's sampling points as the feature vector of the audio signal.

[0010] CN107742524A discloses a signal analysis system that saves time. CN107742524A processes the same frame simultaneously using two paths. It includes a generation module that preprocesses the initial audio signal to generate two audio signals identical to the preprocessed initial audio signal: the two audio signals are a first audio signal and a second audio signal; a preliminary selection module for initial frequency selection of the first audio signal; a feature extraction module for extracting features from the second audio signal; a standardization module for standardizing the features extracted by the feature module; and a weighted average module for calculating the fundamental frequency estimate of the initial audio signal using the standardized features and the initially selected frequencies. By setting up these modules, the initial frequency selection and feature extraction are performed separately and simultaneously, and the fundamental frequency estimate is obtained by weighting the audio signal.

[0011] US20110029306A1 discloses an audio signal recognition apparatus and method. It uses multiple optional stages to process a frame. The input audio signal is identified as either a speech signal or a non-speech signal using at least one feature parameter, and the recognition result of the audio recognizer determines whether to drive an audio discriminator adjacent to the corresponding audio discriminator. This patent application proposes several schemes to improve the accuracy of the recognizer, or to save time or power consumption. First, a simple feature determiner is used. If the result of the feature determiner is positive, a more complex feature determiner is used. This scheme is useful if the probability of a positive result from the simple feature determiner is very low.

[0012] US20180108345A1 discloses an apparatus and method for processing audio signal frames. This patent application uses general processing for all frames, while performing additional processing on some frames as needed. An estimate is calculated based on first-order scattering features to determine whether the first-order scattering features contain sufficient information for accurate audio signal recognition. Second-order scattering features are calculated only if the first-order scattering features do not contain sufficient information for accurate audio signal recognition. The approach in US20180108345A1 is similar to that in US20110029306A1.

[0013] However, no existing technology considers the impact of the input audio signal on the accuracy of the speech recognition result to differentiate the importance of the audio signal. If audio signals could be differentiated according to their importance to the recognition result, and then more comprehensive and complex processing could be applied to important audio signals, while less important audio signals could be simplified, this approach could potentially reduce the computational load of feature extraction processing while maintaining the accuracy of the recognition result based on the extracted features. Summary of the Invention

[0014] To achieve the above objectives, in addition to simplifying the feature extraction path in the prior art, embodiments of this disclosure propose to distinguish the importance of audio signals based on the salience values ​​of audio signal frames in the estimated time series, and then dynamically select different feature extraction processes according to the different importance of the audio signals to distinguish and process the corresponding audio signals to obtain feature vectors for the recognition engine to train and perform speech recognition.

[0015] According to a first aspect of this disclosure, a method for processing an audio signal is provided, comprising: preprocessing an input audio sequence signal within a time period, including dividing the audio sequence signal within the time period into multiple audio signal frames; classifying the multiple audio signal frames into at least important frames or unimportant frames based on at least the saliency values ​​of the multiple audio signal frames; performing feature extraction on a first proportion of the audio signal in each important frame using a high-level feature extraction process; performing feature extraction on a second proportion of the sampled audio signal in each unimportant frame using a low-level feature extraction process, wherein the second proportion is less than the first proportion and the second proportion is greater than zero; and at least matching and integrating the features extracted from the important frames and the features extracted from the unimportant frames to generate a feature vector for the audio sequence signal within the time period.

[0016] According to a second aspect of this disclosure, an audio signal processing system includes: a preprocessing unit for dividing an input audio sequence signal within a time period into multiple audio signal frames; a saliency classification unit for classifying the multiple audio signal frames into at least important frames or unimportant frames based on at least the saliency values ​​of the multiple audio signal frames; a high-level feature extraction unit for performing feature extraction on a first proportion of the audio signal in each important frame; a low-level feature extraction unit for performing feature extraction on a second proportion of the audio signal in each unimportant frame, wherein the second proportion is less than the first proportion and the second proportion is greater than zero; and a feature integration unit for matching and integrating at least the features extracted from the important frames and the features extracted from the unimportant frames to generate a feature vector for the audio sequence signal within the time period.

[0017] According to a third aspect of this disclosure, a computer-readable storage medium is provided having a computer program stored thereon that, when run on a computing device, causes the computing device to perform the method of processing audio signals according to this disclosure.

[0018] These and other aspects of this disclosure will be apparent from the embodiments described below, and will be elucidated with reference to the embodiments described below. Attached Figure Description

[0019] Further details, features, and advantages of this disclosure are disclosed in the following description of exemplary embodiments in conjunction with the accompanying drawings, in which: Figure 1An exemplary system 100 for extracting audio signal features according to an embodiment of the present disclosure is illustrated schematically; Figure 2 This schematically illustrates a basic flowchart 200 of an exemplary embodiment of the present disclosure for dynamically selecting different time-frequency conversion methods based on the importance of audio signal frames; Figure 3 This schematically illustrates a specific flowchart 300 of an exemplary embodiment of the present disclosure for dynamically selecting different time-frequency conversion methods based on the importance of audio signal frames; Figure 4 A block diagram 400 schematically illustrates a time-frequency conversion method in a feature extraction process of three different levels of complexity, according to embodiments of the present disclosure. Figure 5 schematically illustrates two methods for feature matching processing after different levels of time-frequency conversion according to embodiments of the present disclosure; Figure 6 A block diagram 600 schematically illustrates the optimized threshold estimation method; Figure 7 An exemplary block diagram 700 is schematically shown according to a first specific embodiment of the present disclosure; Figure 7A The illustration schematically depicts the conversion from an audio signal to a time-frequency characteristic spectrum implemented according to a first specific embodiment of the present disclosure; Figure 7B A flowchart illustrating an exemplary high-salience region analysis process according to a first specific embodiment of the present disclosure is shown schematically. Figure 8 A schematic flowchart 800 is shown according to a second specific embodiment of the present disclosure; Figure 8A A feature vector map obtained according to the method of a second specific embodiment of this disclosure is illustrated schematically; Figure 9 The test results show the computational complexity of obtaining feature vectors using different preset thresholds and the accuracy of the recognition results obtained based on the feature vectors according to the first and second embodiments of this disclosure. Figure 10 The functional modules of the existing MFCC are shown; Figure 11 A block diagram of a computing device 1100 according to an embodiment of the present disclosure is shown schematically. Detailed Implementation

[0020] Before detailing the embodiments of this disclosure, some related concepts will be explained. As previously mentioned, some embodiments of this disclosure propose distinguishing the importance of audio signals based on the degree of influence of the input audio signal on the accuracy of speech recognition results, and dynamically adopting different processing methods according to the importance of the audio signal to perform feature extraction on the corresponding audio signal. Regarding how to determine the importance of an audio signal, embodiments of this disclosure propose estimating the importance of the audio signal based on the estimated saliency value of the audio signal frame. The estimation of the saliency value of the audio signal frame can be achieved by calculating various characteristics of the audio signal. For example, the saliency value of each audio signal frame can be estimated by calculating characteristics such as the amplitude, power, zero-crossing rate in the time domain, STFT, wavelet, MFCC, and signal-to-noise ratio of the audio signal.

[0021] The embodiments of this disclosure provide a scheme for dynamically employing different processing procedures for feature extraction based on the importance of the audio signal, which is specifically illustrated through the following embodiments.

[0022] Figure 1 The illustration schematically depicts an exemplary system 100 for dynamically extracting audio signal features based on the importance of audio signal frames according to embodiments of the present disclosure. Various methods described herein can be implemented in this system 100. For example... Figure 1 As shown, the system 100 for dynamically extracting audio signal features includes a preprocessing unit 101, an importance classification unit 102, a high-level feature extraction unit 103-1, a low-level feature extraction unit 103-2, a feature matching unit 104, a saliency value estimation unit 106, and a threshold estimation unit 107. The system 100 may optionally also include multiple processing units, such as a log unit 108, a merging and differencing processing unit 109, etc.

[0023] Figure 2 A flowchart 200 is illustrated schematically of an exemplary basic method for dynamically selecting different time-frequency transitions based on the importance of audio signal frames, according to an embodiment of the present disclosure.

[0024] At step 201, the input audio file, which comprises an audio sequence signal over a time period, is preprocessed. Preprocessing of the input audio sequence signal may include noise filtering, pre-emphasis, framing, windowing, VAD, etc. Framing here refers to dividing the audio sequence signal within the time period into multiple audio signal subframes. The duration of each subframe can be 20-40 ms, as needed. Each frame of audio signal should be relatively stable and have sufficient samples for evaluation. In some embodiments, overlap between multiple audio signal frames may be permitted.

[0025] At step 202, a saliency analysis is performed on the preprocessed audio signal. The saliency value of each audio signal frame can be estimated by calculating the amplitude, power, zero-crossing rate in the time domain, STFT, wavelet, MFCC, signal-to-noise ratio, etc.

[0026] In step 203, the importance of the current frame is determined based on the estimated saliency values ​​of one or more audio signal frames, and a different level of feature extraction process is dynamically selected accordingly to extract features from the current frame. For example, a high-level feature extraction process can be used to process current frames determined to be important frames, while a low-level feature extraction process can be used to process current frames determined to be unimportant frames. In some embodiments, during the high-level feature extraction process, a higher proportion of the audio signal can be selected for FFT processing as needed during time-frequency conversion, and subsequent feature extraction can be performed on the higher proportion of the audio signal accordingly. Conversely, during the low-level feature extraction process, a lower proportion of the audio signal is selected for FFT processing, and subsequent feature extraction can be performed on the lower proportion of the audio signal accordingly.

[0027] In step 204, after performing time-frequency conversion on audio signals of different proportions for important and unimportant frames, various methods can be used to process the frequency domain signal to obtain feature vectors as needed. Since feature extraction, such as MFCC, essentially calculates the energy of the frequency domain signal, for relatively simple applications, feature vectors can be obtained directly based on the amplitude or energy of the frequency domain signal. In another embodiment, one or more different modules after the FFT module 1002 in the conventional MFCC feature extraction process described above can be selected to process and obtain feature vectors, depending on the needs of different applications. For example, the Mel spectrum can be obtained through the Mel filter bank and / or the logarithm can be taken using the Log module. Furthermore, delta can be added as needed to transform a time-frequency feature vector into multiple feature vectors, and so on.

[0028] In addition to selecting different proportions of audio signals for important and unimportant frames during time-frequency conversion at step 203, some embodiments may also employ different processing methods for important and unimportant frames at preprocessing step 201 and step 204 after time-frequency conversion.

[0029] In one embodiment, in preprocessing step 201, different windowing processes can be applied to important frames and unimportant frames. For example, for unimportant frames where a smaller proportion of the audio signal is processed during time-frequency conversion, a different window function, such as a shorter one, can be used for windowing compared to important frames. Since the purpose of windowing is to reduce spectral energy leakage during time-frequency conversion, using a shorter window function for windowing unimportant frames can better reduce spectral energy leakage during time-frequency conversion of unimportant frames. In another embodiment, the same windowing process can be applied to both important and unimportant frames, i.e., the same window function can be used for both, which will correspondingly increase spectral energy leakage during time-frequency conversion of unimportant frames.

[0030] Figure 3 A detailed flowchart 300, according to an embodiment of the present disclosure, illustrates an exemplary method for dynamically selecting different time-frequency conversion methods based on the importance of an audio signal frame. Figure 3 The illustrated embodiment is performed on multiple frames of audio signals after preprocessing the input speech signal.

[0031] At step 302, saliency values ​​for one or more audio signal frames are estimated to assess the importance of the current frame. The determination of saliency values ​​can be based on time-domain features, such as time-domain amplitude, power / energy, zero-crossing rate, signal-to-noise ratio, etc.; or on frequency-domain features, such as frequency-domain amplitude, power / energy, power / energy in the frequency band, MFSC, MFCC results, etc. In one embodiment, saliency values ​​can be estimated before feature extraction. Figure 2 After preprocessing step 201, the saliency values ​​of each frame of the audio signal are estimated and stored. Therefore, during feature extraction, any one or more audio signal frames can be extracted and processed at any time.

[0032] At step 303, the importance of the current frame is determined. The importance of the current frame can be determined using characteristics of different audio signal frames in the input speech signal. For example, it can be determined using the salience value of the current frame estimated in step 302, predicted using the salience values ​​of one or more previous audio signal frames, or determined using one or more subsequent audio signal frames. Using previous or subsequent audio signal frames to determine the salience level of the current frame eliminates the problem of potentially misjudging the salience level of the current frame due to its low power / energy. In one instance, the current frame can be classified as at least important or unimportant by comparing its salience value with one or more thresholds, comparing the salience values ​​of one or more previous audio signal frames with one or more thresholds, or comparing the salience values ​​of one or more subsequent audio signal frames with one or more thresholds, allowing for different processing in subsequent steps.

[0033] In steps 304 and 305, different levels of time-frequency feature extraction processes are performed on audio signal frames with different levels of importance. In one embodiment, the current frame can be classified as an important frame or a non-important frame, with a higher-level feature extraction process applied to important frames and a lower-level feature extraction process applied to non-important frames. In general, a higher-level feature extraction process can obtain more accurate feature vectors, but the process is more complex and involves a greater amount of computation.

[0034] Figure 4 A time-frequency conversion method 400 is illustrated in three exemplary feature extraction processes with different levels of complexity according to embodiments of the present disclosure. In this embodiment, three different levels of processing are employed when performing an FFT transform on the current frame, based on three different ranges of salience values ​​of the current, previous, or subsequent audio signal frames. For the audio signal frame 401 with the highest salience value range of 1, the complete frame information of the current frame is used for FFT processing, for example, all 256 sample points are processed using FFT. For different applications, different proportions of sample point information can be selected as needed to perform FFT processing on the audio signal frame with the highest salience value range of 1, for example, half to all of the information of all sample points can be used for FFT processing.

[0035] For an audio signal frame 402 with a moderate saliency value range of 2, half of the frame information of the current frame can be selected, i.e., 128 sampling points can be used for FFT processing. For different applications, sampling point information of, for example, 1 / 4 to 1 / 2 of the audio signal frame with a moderate saliency value range of 2 can be selected for FFT processing as needed.

[0036] For audio signal frame 403 with a low significance value range of 3, only 1 / 4 of the sampling points, i.e., 64 sampling points, can be selected for FFT processing. Since FFT processing requires a significant amount of computation, performing FFT processing on 64 sampling points is much simpler in terms of computational complexity than FFT processing on 256 points. For audio signal frames with a low significance value range of 3, for example, 1 / 8 to 1 / 4 of the sampling point information can be selected for FFT processing. After performing FFT processing, the corresponding spectrum obtained after time-frequency conversion can be further processed by methods such as Mel filtering and cepstral analysis to obtain feature vectors.

[0037] The inventors of this application have discovered that the accuracy of the recognition result obtained through processing and analysis by a recognition engine based on the obtained feature vector is not positively correlated with the signal volume or computational complexity of the sample points of the processed audio signal frames. In other words, if all audio signal frames are processed using the highest level of complexity, recognition processing based on the obtained feature vector does not yield the most accurate recognition result. For example, if all audio signal frames of the input speech signal are processed using complete frame information for each frame via FFT, the accuracy of the recognition result obtained based on the feature vector is actually lower than that based on... Figure 4 The accuracy of the recognition results obtained by performing FFT on feature vectors from audio signal frames with sampling points of 1, 1 / 2, and 1 / 4 respectively for the three saliency ranges 401, 402, and 403 is low. One reason may be that audio signal frames with low saliency ranges, i.e., the least important frames, contain relatively higher environmental noise. Therefore, processing a smaller proportion of information from unimportant frames reduces the proportion of noise introduced into the obtained feature vectors. The inventors also found that even for audio signal frames with the lowest saliency range, i.e., the least important frames, a certain proportion of sampling points need to be selected for processing during feature extraction, rather than discarding them all. If a portion of frames classified as unimportant are discarded without processing, the accuracy of the final recognition result will be lower compared to the feature vector obtained by selecting a low proportion of sampling points. Through multiple experiments, the inventors of this application have verified that using the feature vectors obtained by selecting different proportions of audio information based on the importance of audio signal frames, as proposed in this disclosure, leads to the recognition results with the highest accuracy.

[0038] Next, in step 306, the features after time-frequency transformation are matched and integrated. Because different levels of time-frequency transformation processing are used in the time-frequency transformation as described above, the amount of information in the frequency domain obtained after different levels of time-frequency transformation processing is not matched, for example... Figure 4 The number of frequency bins output after the three different levels of FFT processing in the embodiments is different. Therefore, matching processing is required for the obtained frequency domain information. Figure 5 illustrates two feature matching processes after different levels of time-frequency conversion according to an embodiment of this disclosure. One method is to merge important frames with more frequency domain information, combining information from some frequency bands together so that the amount of information in the processed important frames remains the same as the amount of information in the non-important frames. Figure 5A This demonstrates how filter banks can be used to combine information from some frequency bands of important frames with more bins, in order to maintain consistency with the amount of frequency domain information in less important frames with fewer bins. Figure 5BAnother method shown is to interpolate in less important frames with fewer bins so that the number of bins in the interpolated less important frames matches the number of bins in the important frames. For example, if the number of bins in a less important frame is 32, interpolation can achieve the same number of bins as the important frames, which is 128.

[0039] In step 307, after matching the information obtained by the feature extraction process at different levels, the feature vector of the input speech signal segment can be generated.

[0040] In step 308, one or more threshold estimates are determined for the frame importance determination in step 303. The threshold estimation can be performed using methods such as... Figure 6 The experimental method shown is to find the optimal threshold for judging the importance of frames. For example, multiple different thresholds 601 can be set to train the mathematical model of speech recognition 602, and the indicators of recognition ability, FRR (False Rejection Rate) and FAR (False Acceptance Rate), can be used as the ordinate and abscissa respectively to generate a curve indicating recognition accuracy. Based on the comparison of recognition accuracy and computational complexity 603, the optimal threshold 604 is selected. Here, FRR refers to the false rejection rate, that is, the ratio of frames that should have passed recognition but did not, while FAR refers to the false acceptance rate, that is, the ratio of frames that should have failed recognition but did.

[0041] Regarding how to set the threshold for judging frame importance, the inventors of this application discovered that the computational complexity of feature extraction is roughly positively correlated with the saliency threshold, with computational complexity increasing as the saliency threshold increases. However, the accuracy of the final recognition result obtained based on the extracted features is not necessarily always positively correlated with the saliency threshold used in the feature extraction process. The inventors of this application discovered that as the saliency threshold used for feature extraction increases, the accuracy of the recognition result obtained based on the extracted features will reach a peak. After this peak, as the saliency threshold used for feature extraction increases, the accuracy of the recognition result obtained based on the extracted features will actually decrease. Therefore, the optimized threshold is selected based on a preset threshold corresponding to the peak value of the accuracy of the recognition result obtained based on the extracted features. In the following specific embodiments, experimental results on how to select the optimized threshold based on the peak value of the accuracy of the recognition result obtained based on the extracted features are given.

[0042] The following provides an exemplary first specific embodiment based on the inventive concept of this disclosure.

[0043] The first specific embodiment distinguishes important frames from unimportant frames based on the power / energy in the time domain of the audio signal frame.

[0044] Figure 7 An exemplary flowchart according to a first specific embodiment of the present disclosure is shown. This embodiment is based on the Short Time Fourier Transform (STFT) of frame power, without windowing and Mel-filter processing. After preprocessing the input speech signal 701 and framing 702, the saliency value 703 of each audio signal frame is estimated. Figure 7 The image further illustrates the first and second frames 711 after framing, and the first and second frames may include an overlapping portion.

[0045] In this embodiment, audio signal frames are divided into two levels based on their importance: important frames and unimportant frames, according to the salience value of the current frame. For important frames, a high-level feature extraction process 700A is used for feature extraction. In the FFT processing step 704A of the high-level feature extraction process, FFT processing is performed on all 256 sample points of the important frame. Then, in step 705A, the power / energy of the FFT-transformed result is calculated, for example, based on the frequency domain signal amplitude or frequency domain signal energy. In step 706A, a filter bank, such as a Mel filter bank, is used to filter the information of the 256 sample points in the time domain into 32 feature signals in the frequency domain. In step 706A, the power of each frequency bin in each equal frequency band is averaged. For non-critical frames, a low-level feature extraction process 700B is used for feature extraction. In the FFT processing step 704B of the low-level feature extraction process, only 1 / 4 of the sampled information in the non-critical frame is selected, while the remaining sampled information in the non-critical frame is discarded. That is, information from 64 sample points is used for FFT processing. Then, in step 705B, the power / energy of the FFT transformation result is calculated, and 32 feature signals in the frequency domain are obtained. Then, in step 707, the time-frequency transformed information of critical and non-critical frames is processed by Log. Since information from multiple frames is needed for subsequent speech recognition, in step 708, the 32 frequency domain information from each frame of multiple frames needs to be merged. For example, the frequency domain information of 32 frames is merged to form a feature map 710 of 32 feature information × 32 frames. In this example feature map 710, the horizontal axis represents the time domain, and the vertical axis represents the frequency domain. A certain time point in the time domain has a corresponding 32 frequency domain signals. The lighter-colored portion mainly consists of 32 frequency domain signals from important frames processed by an FFT of 256 sampling points per frame, while the darker-colored portion mainly consists of 32 frequency domain signals generated from non-important frames processed by an FFT of 64 sampling points per frame. In step 709, adding delta transforms one time-frequency feature vector into multiple feature vectors. It can be understood that in different applications, after performing time-frequency transformations on important and non-important frames in steps 704A and 704B respectively, various processing methods can be applied to the frequency domain signals to obtain time-frequency feature vectors as needed. For example, as described above... Figure 2 Various different methods for obtaining feature vectors are performed at step 204.

[0046] Figure 7A The illustration schematically depicts the conversion from an audio signal to a time-frequency characteristic spectrum implemented according to a first specific embodiment of the present disclosure. Figure 7A The diagram clearly shows the correspondence between the high-level and low-level feature extraction processes applied to key and non-key frames in an audio signal, and the resulting feature spectra.

[0047] Optionally, the step of estimating the salience value 703 for each audio signal frame may include an analysis process for highly salience regions, in which the importance of frames may be determined using multiple audio signal frames through a MAX filter. Figure 7B A flowchart of the analysis process for highly significant regions is shown. Figure 7B Figure 7036 shows an audio signal segment with high-salience and low-salience regions. In step 7031, the power of multiple time-domain frames is calculated. In step 7032, a MAX filter is used to reassign the low-power frames within the high-salience regions. The use of the MAX filter eliminates the possibility of misjudging the importance of the current frame due to its low energy. Figure 7B As shown in step 7035, there may be a situation where the amplitude values ​​of the two frames before and after the current frame are both high, but the amplitude value of the current frame is low. In this case, the MAX filter can be used to reassign the amplitude values ​​of the current frame in the middle by referring to the amplitude values ​​before and after it. In this case, even if the amplitude value of the current frame does not exceed the threshold, it cannot be considered an unimportant frame. After the MAX filter, in step 7033, the current frame is determined to be an important frame or an unimportant frame by comparing it with the threshold.

[0048] Figure 8 A flowchart 800 is shown as an example of a second specific embodiment of the present disclosure.

[0049] In this second specific embodiment, saliency analysis is performed based on the frequency domain characteristics of the audio signal to distinguish between important and unimportant frames. In this embodiment, the features of the previous frame are used to determine the importance of the current frame. After pre-emphasis and framing processing of the input speech signal in steps 801 and 802, the first frame is defaulted to an unimportant frame. In step 804A, 1 / 4 of the sampling information of the first frame, i.e., 64 sampling points, is selected for FFT processing. Then, in step 805A, the power / energy of the FFT result is calculated, and in step 806, the low-frequency power of the first frame is calculated. Next, returning to step 803, if the low-frequency power / energy of the first frame is greater than a preset threshold, the next frame is judged as an important frame. Then, in step 804B, FFT processing is performed on the information of all 256 sampling points in the frame. Then, in step 805B, the power / energy of the FFT transformation result is calculated and filtered using a filter bank. After calculating the low-frequency power in step 806, the logarithm of the audio signal energy spectrum in the frequency domain is taken in step 807 to obtain the frequency domain feature information of each frame. The frequency domain feature information of each frame is merged to form a feature vector map. Then, in step 808, the feature vector map is subjected to one or more differential processing. The feature vector map and the result of the differential processing are used as input to the recognition engine for subsequent processing.

[0050] In this embodiment, since low-frequency information is more important, it is used for frame importance analysis. Furthermore, since the importance of the current frame is determined by using the low-frequency power of the previous frame calculated in the frequency domain, there is no need to calculate the time-domain power, thus further reducing the computational load. Figure 8A The low-frequency portion of the feature vector map obtained according to the method of the second embodiment of this disclosure is shown.

[0051] Figure 9 The test results of the computational complexity of obtaining feature vectors using different saliency thresholds and the accuracy of the recognition results obtained based on the feature vectors are shown according to the first and second embodiments of this disclosure. Figure A shows the test results of the computational complexity of obtaining feature vectors using different thresholds and the accuracy of the recognition results obtained based on the feature vectors according to the first embodiment. Five keywords (sheila, marvin, bird, dog, cat) were used for testing. Figure B shows the test results of the computational complexity of obtaining feature vectors using different thresholds and the accuracy of the recognition results obtained based on the feature vectors according to the second embodiment. Five keywords (1, 2, 3, 4, 5) were used for testing. From the test results in Figures A and B, it can be seen that the computational complexity of obtaining feature vectors is roughly positively correlated with the saliency threshold, and the computational complexity increases with the increase of the saliency threshold. However, the accuracy of the recognition results does not necessarily always show a positive correlation with the saliency threshold. As the saliency threshold increases, the accuracy of the recognition results will show a peak. After the peak, further increases in the saliency threshold will actually decrease the accuracy of the recognition results. In the example of Figure A, the accuracy of the recognition result reaches its peak when the saliency threshold is set to 1e6, so the saliency threshold of 1e6 can be set as the preferred threshold. In the example of Figure B, the accuracy of the recognition result reaches its peak when the saliency threshold is set to 1e8, so the saliency threshold of 1e8 can be set as the preferred threshold.

[0052] Although the feature extraction in both embodiments of this disclosure is performed on digital signals in the digital domain, feature extraction can also be performed in the analog domain. For example, a conventional method for analog feature extraction uses a set of bandpass filters to decompose the signal in the frequency domain. Those skilled in the art will understand that the basic inventive concept of this disclosure, which uses the saliency values ​​of audio signal frames to distinguish the importance of audio signals and then dynamically selects different features based on the different importance levels of the audio signals, is also applicable to feature extraction in the analog domain.

[0053] Figure 11A block diagram of a computing device 1100 according to an embodiment of the present disclosure is shown schematically. The computing device 1100 represents... Figure 1 The system 100 for dynamically extracting audio signal features includes a preprocessing unit 101, an importance classification unit 102, a high-level feature extraction unit 103-1, a low-level feature extraction unit 103-2, a feature matching unit 104, a saliency value estimation unit 106, and a threshold estimation unit 107. Optionally, the system 100 may also include multiple processing units, such as one or more log units 108, merging and differencing processing units 109.

[0054] The computing device 1100 can be of various types, such as a server computer, a device associated with a client (e.g., a client device), a system-on-a-chip, and / or any other suitable computing device or computing system.

[0055] The computing device 1100 may include at least one processor 1102, memory 1104, at least two communication interfaces 1106, display device 1108, other input / output (I / O) devices 1110, and one or more mass storage devices 1112 that are capable of communicating with each other, such as via a system bus 1114 or other suitable means.

[0056] Processor 1102 may be a single processing unit or at least two processing units, and all processing units may include a single or at least two computing units or at least two cores. Processor 1102 may be implemented as one or more microprocessors, microcomputers, microcontrollers, digital signal processors, central processing units, state machines, logic circuits, and / or any device that manipulates signals based on operating instructions. Among other capabilities, processor 1102 may be configured to acquire and execute computer-readable instructions stored in memory 1104, mass storage device 1112, or other computer-readable media, such as program code of operating system 1116, program code of application program 1118, program code of other program 1120, etc., to implement the method for processing audio signals provided in embodiments of this disclosure.

[0057] Memory 1104 and mass storage device 1112 are examples of computer storage media for storing instructions that are executed by processor 1102 to perform the various functions described above. For example, memory 1104 may generally include both volatile and non-volatile memory (e.g., RAM, ROM, etc.). Furthermore, mass storage device 1112 may generally include hard disk drives, solid-state drives, removable media (including external and removable drives), memory cards, flash memory, floppy disks, optical disks (e.g., CDs, DVDs), storage arrays, network storage, storage area networks, etc. Both memory 1104 and mass storage device 1112 may be collectively referred to herein as memory or computer storage media, and may be non-transitory media capable of storing computer-readable, processor-executable program instructions as computer program code that can be executed by processor 1102 as a specific machine configured to perform the operations and functions described in the examples herein.

[0058] At least two program modules may be stored on mass storage device 1112. These programs include operating system 1116, one or more application programs 1118, other programs 1120, and program data 1122, and they may be loaded into memory 1104 for execution. Examples of such application programs or program modules may include, for example, computer program logic (e.g., computer program code or instructions) for implementing the various processing units of this disclosure.

[0059] Although Figure 11 The modules 1116, 1118, 1120, and 1122, or portions thereof, are illustrated as being stored in memory 1104 of computing device 1100; however, modules 1116, 1118, 1120, and 1122 may be implemented using any form of computer-readable medium accessible by computing device 1100. As used herein, “computer-readable medium” may include one or more types of computer-readable media, such as computer storage media and / or communication media.

[0060] Computer storage media includes volatile and non-volatile, removable and non-removable media implemented by any method or technology for storing information, such information as computer-readable instructions, data structures, program modules, or other data. Computer storage media includes, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technologies, CD-ROM, DVD, or other optical storage devices, magnetic cartridges, magnetic tapes, disk storage devices or other magnetic storage devices, or any other non-transfer medium that can be used to store information for access by computing devices.

[0061] In contrast, communication media can embody computer-readable instructions, data structures, program modules, or other data within modulated data signals such as carrier waves or other transmission mechanisms. Computer storage media as defined herein do not include communication media.

[0062] The computing device 1100 may also include one or more communication interfaces 1106 for exchanging data with other devices, such as via a network, direct connection, etc. The communication interface 1106 can facilitate communication across various network and protocol types, including wired networks (e.g., LAN, cable, etc.) and wireless networks (e.g., WLAN, cellular, satellite, etc.), the Internet, etc. The communication interface 1106 can also provide communication with external storage devices (not shown), such as storage arrays, network storage, storage area networks, etc.

[0063] In some examples, a display device 1108, such as a monitor, may be included for displaying information and images. Other I / O devices 1110 may be devices that receive various inputs from the user and provide various outputs to the user, and may include touch input devices, gesture input devices, cameras, keyboards, remote controls, mice, printers, audio input / output devices, and so on.

[0064] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, functional units, and modules described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0065] By studying the accompanying drawings, the disclosure, and the appended claims, those skilled in the art can understand and implement variations of the disclosed embodiments in practicing the claimed subject matter. In the claims, the words "A and / or B" refer to A, B, or A and B; the word "comprising" does not exclude other elements or steps; and the indefinite articles "a" or "an" do not exclude a plurality; the words "first," "second," "third," and "fourth" are used merely to distinguish elements or steps and do not indicate the order of the elements or steps. The mere fact that certain measures are recited in mutually different dependent claims does not indicate that a combination of these measures cannot be beneficial.

Claims

1. A method for processing audio signals, comprising: Preprocessing the input audio sequence signal within a certain time period includes dividing the audio sequence signal within that time period into multiple audio signal frames; Classifying each of the plurality of audio signal frames into at least important frames or unimportant frames based on at least one or more saliency values ​​of the audio signal frames; including: estimating the saliency value of the audio signal frame based on at least one of the amplitude, energy, zero-crossing rate, and signal-to-noise ratio of the sampled audio signal of each audio signal frame in the time domain or frequency domain. A high-level feature extraction process is used to perform feature extraction on the first proportion of the audio signal in each important frame; Feature extraction is performed on the audio signal of a second proportion in each non-important frame using a low-level feature extraction process, wherein the second proportion is less than the first proportion and the second proportion is greater than zero; At least the features extracted from important frames and the features extracted from unimportant frames are matched and integrated to generate a feature vector for the audio sequence signal within that time period. The preprocessing of the audio sequence signal also includes applying different window functions to important and unimportant frames, with the window function applied to unimportant frames being shorter than that applied to important frames.

2. The method as described in claim 1, wherein, The step of classifying multiple audio signal frames also includes classifying the multiple audio signal frames into moderately important frames, and performing feature extraction on the audio signal of a third proportion in each moderately important frame using a medium-level feature extraction process, wherein the third proportion is less than the first proportion and greater than the second proportion.

3. The method as described in claim 1, wherein, The first ratio ranges from 1 / 2 to all features of the frame audio signal, the second ratio ranges from 1 / 8 to 1 / 4, and the third ratio ranges from 1 / 4 to 1 / 2.

4. The method of claim 3, wherein, Feature extraction is performed on a second proportion of sampled audio signals in each non-important frame using a low-level feature extraction process, and the remaining sampled audio signals in the non-important frames are discarded.

5. The method of claim 1, wherein, The current audio signal frame is classified as an important frame or an unimportant frame based on one of the following comparison results: The significance value of the current audio signal frame is compared with at least one threshold; A comparison of the significance values ​​of one or more previous audio signal frames with at least one threshold; or The significance value of one or more subsequent audio signal frames is compared with at least one threshold.

6. The method of claim 5, wherein, The current audio signal frame is classified as at least important or unimportant based on a comparison of the saliency value of a previous audio signal frame in the frequency domain with at least one threshold, wherein the saliency value of the audio signal frame is determined based on the energy of the low-frequency portion of the previous audio signal frame in the frequency domain, and wherein the first frame of the plurality of audio frames is defaulted to an unimportant frame, and a low-level feature extraction process is used to calculate the energy of the low-frequency portion of the first frame and determine the saliency value of the first frame.

7. The method of claim 5, further comprising the step of determining at least one optimization threshold, including: Multiple saliency preset thresholds are applied to the speech model for training to estimate the computational complexity of extracting features using each preset threshold, and to estimate the accuracy of the recognition result obtained by the recognition engine based on the extracted features. A preset threshold corresponding to the peak value of the accuracy of the recognition result is selected as the optimization threshold, wherein the accuracy of the recognition result and the significance threshold are not positively correlated.

8. The method of claim 7, wherein, The accuracy of the recognition result obtained using each preset threshold is estimated based on indicators of recognition capability, including false rejection rate (FRR) and false acceptance rate (FAR), and the computational complexity of extracting features using each preset threshold is estimated based at least on the number of butterfly units in the FFT.

9. The method of claim 1, wherein, The high-level feature extraction process includes performing a Fast Fourier Transform (FFT) on the sampled audio signal of a first proportion in each important frame, calculating the energy spectrum of the audio signal in the frequency domain after the FFT, and applying a band filter to each band of the energy spectrum.

10. The method of claim 9, wherein, The low-level feature extraction process involves performing a Fast Fourier Transform (FFT) on the sampled audio signal at a second proportion in each non-important frame, and calculating the energy spectrum of the audio signal in the frequency domain after the FFT.

11. The method of claim 10 further includes taking the logarithm (LOG) of the energy spectrum of the audio signal in the frequency domain of the obtained important frames and unimportant frames to obtain the frequency domain feature information of each frame, merging the frequency domain feature information of each frame to form a feature vector map, performing one or more differential processing on the feature vector map, and using the feature vector map and the result of the differential processing together as input to the recognition engine for subsequent processing.

12. The method of claim 1, wherein, The multiple audio signal frames that are divided may include overlapping portions.

13. The method of claim 2, wherein, The high-level feature extraction process, the mid-level feature extraction process, and the low-level feature extraction process include time-frequency transformation (TF) feature extraction.

14. An audio signal processing system, comprising: The preprocessing unit is configured to divide the input audio sequence signal within a time period into multiple audio signal frames; The importance classification unit is configured to classify each of the multiple audio signal frames as important or unimportant frames based on the saliency values ​​of at least one or more audio signal frames; it also includes a saliency value estimation unit configured to estimate the saliency value of the audio signal frame based on at least one of the amplitude, energy, zero-crossing rate, and signal-to-noise ratio of the audio signal of each audio signal frame in the time domain or frequency domain. The high-level feature extraction unit is configured to perform feature extraction on a first proportion of the audio signal in each important frame; The low-level feature extraction unit is configured to perform feature extraction on the audio signal of a second proportion in each non-important frame, wherein the second proportion is less than the first proportion and the second proportion is greater than zero. The feature integration unit is configured to match and integrate at least the features extracted from important frames and the features extracted from non-important frames to generate a feature vector for the audio sequence signal within the time period. The preprocessing unit is further configured to apply different window functions to important frames and unimportant frames, wherein the window function applied to unimportant frames is shorter than the window function applied to important frames.

15. The system of claim 14, wherein, The importance classification unit is further configured to classify multiple audio signal frames into medium importance frames, and the system further includes a medium-level feature extraction unit configured to perform feature extraction on a third proportion of the audio signal in each medium importance frame, wherein the third proportion is less than the first proportion and greater than the second proportion.

16. The system of claim 15, wherein, The first ratio ranges from 1 / 2 to all features of the frame audio signal, the second ratio ranges from 1 / 8 to 1 / 4, and the third ratio ranges from 1 / 4 to 1 / 2.

17. The system of claim 16, wherein, The low-level feature extraction unit is also configured to perform feature extraction on a second proportion of audio signals in each non-important frame and discard the remaining audio signals in the non-important frames.

18. The system of claim 14, wherein, The importance classification unit is also configured to classify the current audio signal frame as an important frame or a non-important frame based on one of the following comparison results: The significance value of the current audio signal frame is compared with at least one threshold; A comparison of the significance values ​​of one or more previous audio signal frames with at least one threshold; or The significance value of one or more subsequent audio signal frames is compared with at least one threshold.

19. The system of claim 18, wherein, The importance classification unit is further configured to classify the current audio signal frame as at least an important frame or an unimportant frame based on a comparison of the salience value of a previous audio signal frame in the frequency domain with at least one threshold, wherein the salience value of the audio signal frame is determined based on the energy of the low-frequency portion of the frequency domain of the previous audio signal frame, and wherein the first frame of the plurality of audio frames is defaulted to an unimportant frame, and the low-level feature extraction unit is further configured to calculate the energy of the low-frequency portion of the first frame and determine the salience value of the first frame.

20. The system of claim 18, further comprising a threshold estimation unit for determining at least one optimized threshold, configured to: Multiple saliency preset thresholds are applied to the speech model for training to estimate the computational complexity of extracting features using each preset threshold, and to estimate the accuracy of the recognition result obtained by the recognition engine based on the extracted features. A preset threshold corresponding to the peak value of the accuracy of the recognition result is selected as the optimization threshold, wherein the accuracy of the recognition result and the significance threshold are not positively correlated.

21. The system of claim 20, wherein, The accuracy of the recognition result obtained using each preset threshold is estimated based on indicators of recognition capability, including false rejection rate (FRR) and false acceptance rate (FAR), and the computational complexity of extracting features using each preset threshold is estimated based at least on the number of butterfly units in the FFT.

22. The system of claim 14, wherein, The high-level feature extraction unit is also configured to perform a Fast Fourier Transform (FFT) on a first proportion of the sampled audio signal in each important frame, calculate the energy spectrum of the audio signal in the frequency domain after the FFT, and apply a band filter to each band of the energy spectrum.

23. The system of claim 22, wherein, The low-level feature extraction unit is also configured to perform a Fast Fourier Transform (FFT) on the sampled audio signal at a second proportion in each non-important frame to calculate the energy spectrum of the audio signal in the frequency domain after the FFT.

24. The system of claim 23 further includes a Log unit configured to take the logarithm LOG of the audio signal energy spectra in the frequency domain of the obtained important frames and unimportant frames to obtain the frequency domain feature information of each frame, and a merging and differential processing unit configured to merge the frequency domain feature information of each frame to form a feature vector map, and to perform one or more differential processing on the feature vector map, and to use the feature vector map and the result of the differential processing together as input to the recognition engine for subsequent processing.

25. The system of claim 14, wherein, The multiple audio signal frames that are divided may include overlapping portions.

26. The system of claim 15, wherein, The high-level feature extraction unit, the mid-level feature extraction unit, and the low-level feature unit are also configured to perform time-frequency transformation (TF) feature extraction.

27. A computer-readable storage medium having a computer program stored thereon that, when run on a computing device, causes the computing device to perform the method according to any one of claims 1 to 13.

Images