Speech keyword recognition method and system suitable for hardware processing
By optimizing the speech keyword recognition method based on Mel-frequency cepstral coefficients and dynamic time warping calculations, the problem of efficient, low-storage real-time recognition for IoT edge devices is solved, achieving high-accuracy speech keyword recognition.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- SHANGHAI JIAOTONG UNIV
- Filing Date
- 2023-08-08
- Publication Date
- 2026-06-02
Smart Images
Figure CN116884396B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of speech keyword recognition technology, specifically to a speech keyword recognition model based on Mel-frequency cepstral coefficients and dynamic time warping suitable for hardware processing, and more particularly to a speech keyword recognition method and system suitable for hardware processing. Background Technology
[0002] Speech, as a vital tool for human communication, is the most natural, fastest, and most convenient means of information exchange between people. Speech recognition technology, as a bridge between humans and machines, holds an unparalleled position and role, irreplaceable by other interaction methods. Speech recognition enables machines to "understand" human language, bringing greater convenience to the automation of machine applications. In recent years, significant achievements have been made in the field of speech recognition research. With continuous improvements in acoustic models, language models, and search algorithms, the performance of recognition systems has been greatly enhanced, providing strong technical support for human-machine interaction and enabling many speech recognition systems to be widely applied in social and economic life.
[0003] Keyword recognition is a subfield of speech recognition, its main purpose being to identify whether specific words appear in a speech signal and their location. Compared to complex image and speech recognition, keyword detection systems have significantly lower algorithmic complexity, making them well-suited for wake-up systems and limited command control systems. In recent years, edge-side keyword recognition has attracted attention. Apple's "Hey Siri" is a mature edge-side keyword wake-up system; it continuously listens to surrounding speech signals, making inferences every 0.2 seconds to determine whether the user has said "Hey Siri," demonstrating high application value.
[0004] Traditional keyword recognition systems typically consist of an acoustic feature extraction module, an acoustic model, a language model, and a decoder. The acoustic feature extraction module preprocesses the audio signal, and the results are obtained by comparing data from the decoder, acoustic model, and language model. Commonly used algorithms include Dynamic Time Warping (VTW), Viterbi algorithm, and Hidden Markov Models (HMMs). Traditional keyword detection systems require complex modular design, and the models limit the recognition range for different human voices. When comparing input data with target words, traditional speech recognition often employs a multi-pass search strategy to ensure accuracy, which improves the recognition rate but also increases latency. For IoT edge node applications, traditional models still have room for improvement in terms of energy efficiency, storage, and accuracy.
[0005] With the rise of neural networks in recent years, they have been applied to keyword recognition. Taking the keyword detection system launched by TensorFlow as an example, the entire system consists of an acoustic feature extraction module and a recognition network. The audio signal is preprocessed into a discrete matrix vector by the acoustic feature extraction module, and then the final result is output after passing through the recognition network. Compared with traditional keyword detection systems, neural network-based keyword detection systems are simpler to design and have relatively higher recognition accuracy. However, the increasingly complex neural networks place high demands on training data and computing power, making it difficult to deploy such recognition systems on edge devices.
[0006] Patent document CN115273891A (application number: CN202210823199.2) discloses an automatic audio error trimming method based on similarity detection. To improve the accuracy of audio error trimming, an audio similarity detection method is proposed. The implementation scheme is as follows: First, dynamic programming is used to obtain speech segments with pre-judged similarity. Based on the pre-judged speech segments, a fast Fourier transform is performed and interpolation is used to obtain the frequency domain similarity between the two speech segments. Furthermore, for the pre-judged speech segments, Mel-frequency cepstral coefficients are calculated and a modified dynamic time warping is performed to obtain the time domain similarity between the two speech segments.
[0007] Traditional speech keyword recognition models based on Mel-Cepstral Coefficients (MCCs) and Dynamic Time Warping (VTW) are relatively mature. The classic steps of MCC extraction and the basic idea of VTW both have their advantages. If only considering application scenarios on personal computers or in the cloud, these models can achieve high accuracy. However, with the increasing demands on edge speech recognition applications in the IoT era, and the higher sampling rate data placing a greater workload on edge applications, speech keyword recognition needs to achieve extremely high energy efficiency, extremely low storage, and high recognition accuracy. This requires innovative techniques to increase data compression and the sufficiency of speech feature extraction, building upon traditional MCC-based speech keyword recognition models, to make keyword recognition more suitable for edge hardware processing.
[0008] Existing methods for speech keyword recognition based on Mel-frequency cepstral coefficients or other speech feature parameters, followed by the use of neural network architectures such as deep neural networks, convolutional neural networks, and recurrent neural networks, typically increase network complexity significantly, require massive amounts of data for training, and deploy a large number of high-performance computing chips for training. These methods aim to further improve keyword recognition accuracy based on algorithms based on dynamic time warping. However, due to the higher demands on computing power and data, these methods are difficult to use in emerging IoT application scenarios with limited training data, edge computing, high energy efficiency, and low storage. Summary of the Invention
[0009] To address the shortcomings of existing technologies, this invention provides a speech keyword recognition method and system suitable for hardware processing.
[0010] According to the present invention, a speech keyword recognition method and system suitable for hardware processing are provided, the solution of which is as follows:
[0011] Firstly, a speech keyword recognition method suitable for hardware processing is provided, the method comprising:
[0012] The speech keyword recognition model recognizes the following steps: Speech is collected and preprocessed, and initial noise reduction and downsampling are performed on high-sampling-rate speech;
[0013] Mel-Cepstral Coefficient Extraction Steps: After initial noise removal and downsampling of high-sampling-rate speech data, Mel-Cepstral coefficients are extracted. Based on the coefficient extraction, mixed-window processing and sparse compression processing are adopted, and sinusoidal spectrum enhancement processing is adopted.
[0014] Dynamic time warping calculation steps: After extracting the Mel-frequency cepstral coefficients of the speech data, dynamic time warping calculation preprocessing is required. Based on the warping path calculation, nonlinear difference processing, data alignment processing, normalization processing, and dual threshold decision are adopted.
[0015] Preferably, in the speech keyword recognition model recognition step, after the high sampling rate speech data is collected and input by the speech acquisition device, initial noise cancellation is first performed to remove the first 5% of the speech data; then, 160-fold downsampling processing is performed to extract 1 data from 160 data points.
[0016] Preferably, in the Mel-Cepstral Coefficient Extraction step, Mel-Cepstral Coefficients are extracted. During the Mel window construction process, partial aliasing is allowed, and the highest frequency of the Mel window is set to 2-6 times that allowed by the sampling theorem. During the Mel window construction process, data sparsity is compressed, and the windowing boundary is recorded to avoid multiplication by zero. After discrete cosine transform processing, a sine lifting window is added.
[0017] Preferably, the dynamic time warping calculation step includes: extracting the Mel-Cepstral coefficients of the speech data, which requires dynamic time warping calculation preprocessing. The preprocessing first uses nearest neighbor interpolation to align the length of the speech data, then performs different nonlinear processing on the two segments of speech data, then performs frame padding on the processed speech data, and finally divides the speech data into blocks of 2-6 frames.
[0018] Preferably, the speech Mel-Cepstral Coefficients obtained by the preprocessing of dynamic time warping calculation are used to perform dynamic time warping calculation between corresponding blocks, and the results of the calculation for corresponding blocks are accumulated to obtain the overall dynamic time warping calculation result.
[0019] Preferably, for the overall dynamic time warping calculation result, two decision values need to be calculated: the original decision value and the normalized decision value;
[0020] The original decision value is equal to the result of the overall dynamic time warping calculation. The normalized decision value is obtained by normalizing the speech length based on the result of the overall dynamic time warping calculation.
[0021] Preferably, a two-level decision is made for the two decision values: direct decision and normalized decision. The direct decision determines whether the speech data contains the corresponding keyword by comparing the original decision value with the maximum and minimum thresholds. If the direct decision fails, a normalized decision is made by comparing the normalized decision value with the degenerate maximum and minimum thresholds to determine whether the speech data contains the corresponding keyword.
[0022] Secondly, a speech keyword recognition system suitable for hardware processing is provided, the system comprising:
[0023] Speech Keyword Recognition Model Recognition Module: Collects speech and performs speech preprocessing, including initial noise reduction and downsampling for high-sampling-rate speech;
[0024] Mel-Cepstral Coefficient Extraction Module: After initial noise removal and downsampling of high-sampling-rate speech data, Mel-Cepstral coefficients are extracted. Based on the coefficient extraction, mixed-window processing and sparse compression processing are adopted, and sinusoidal spectrum enhancement processing is adopted.
[0025] Dynamic time warping calculation module: After extracting the Mel-frequency cepstral coefficients of the speech data, dynamic time warping calculation preprocessing is required. Based on the warping path calculation, nonlinear difference processing, data alignment processing, normalization processing, and dual threshold decision are adopted.
[0026] Preferably, in the speech keyword recognition model recognition module, after the high sampling rate speech data is collected and input by the speech acquisition device, initial noise cancellation is first performed to remove the first 5% of the speech data; then, 160-fold downsampling processing is performed to extract 1 data from 160 data points.
[0027] In the Mel-Cepstral Coefficient Extraction Module, Mel-Cepstral Coefficients are extracted. During the Mel window construction process, partial aliasing is allowed, and the highest frequency of the Mel window is set to 2-6 times that allowed by the sampling theorem. During the Mel window construction process, data sparsity is compressed, and the windowing boundary is recorded to avoid multiplication by zero. After discrete cosine transform processing, a sine lifting window is added.
[0028] Preferably, in the Mel-Cepstral Coefficient Extraction Module, Mel-Cepstral Coefficients are extracted. During the Mel window construction process, partial aliasing is allowed, and the highest frequency of the Mel window is set to 2-6 times that allowed by the sampling theorem. During the Mel window construction process, data sparsity is compressed, and the windowing boundary is recorded to avoid multiplication by zero. After discrete cosine transform processing, a sine lifting window is added.
[0029] The dynamic time warping calculation module includes: extracting Mel-frequency cepstral coefficients from the speech data, which requires dynamic time warping calculation preprocessing. The preprocessing first uses nearest neighbor interpolation to align the length of the speech data, then performs different nonlinear processing on the two segments of speech data, then performs frame padding on the processed speech data, and finally divides the speech data into blocks of 2-6 frames.
[0030] The speech Mel-spectral coefficients of the blocks obtained after dynamic time warping calculation preprocessing are then subjected to dynamic time warping calculation between corresponding blocks. The results of the calculation for each block are then summed to obtain the overall dynamic time warping calculation result.
[0031] For the overall dynamic time warping calculation results, two decision values need to be calculated: the original decision value and the normalized decision value;
[0032] The original decision value is equal to the overall dynamic time warping calculation result. The normalized decision value is obtained by normalizing the speech length based on the overall dynamic time warping calculation result.
[0033] Two levels of decision are performed for the two decision values: direct decision and normalized decision. The direct decision determines whether the speech data contains the corresponding keyword by comparing the original decision value with the maximum and minimum thresholds. If the direct decision fails, the normalized decision is performed by comparing the normalized decision value with the degenerate maximum and minimum thresholds to determine whether the speech data contains the corresponding keyword.
[0034] Compared with the prior art, the present invention has the following beneficial effects:
[0035] 1. This invention does not require a large amount of computing and storage resources, and has higher computing efficiency and less storage resource utilization. For Mel-frequency cepstral coefficient extraction, it reduces computational complexity and greatly reduces speech keyword recognition latency. For dynamic time warping calculation, it improves the nonlinearity between data, improves the decision structure, and improves the accuracy of speech keyword recognition.
[0036] 2. This invention can achieve high accuracy, low storage, and real-time processing of keyword recognition for edge voice, and has a wide range of applications in the construction of current IoT voice nodes.
[0037] Other beneficial effects of the present invention will be explained in detail through the introduction of specific technical features and technical solutions in specific embodiments. Those skilled in the art should be able to understand the beneficial technical effects brought about by these technical features and technical solutions through the introduction of these technical features and technical solutions. Attached Figure Description
[0038] Other features, objects, and advantages of the present invention will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings:
[0039] Figure 1 This is a schematic diagram of the voice keyword recognition system constructed according to the present invention;
[0040] Figure 2 This is a schematic diagram of the Mel-Cepstral coefficient extraction process;
[0041] Figure 3 In Mel cepstral coefficient extraction, during the Mel window construction stage, by allowing appropriate aliasing, the frequency extraction range is made to better match the human speech diagram.
[0042] Figure 4 This is a schematic diagram illustrating the preprocessing of Mel-Cepstral Coefficients features in speech data during dynamic time warping calculations.
[0043] Figure 5 This is a schematic diagram of the overall calculation in dynamic time warping.
[0044] Figure 6 A schematic diagram illustrating the calculation of two decision values in dynamic time warping calculation;
[0045] Figure 7 This is a schematic diagram of a two-level decision process in dynamic time warping calculations. Detailed Implementation
[0046] The present invention will now be described in detail with reference to specific embodiments. These embodiments will help those skilled in the art to further understand the present invention, but do not limit the invention in any way. It should be noted that those skilled in the art can make several changes and improvements without departing from the concept of the present invention. These all fall within the protection scope of the present invention.
[0047] This invention provides a speech keyword recognition method suitable for hardware processing, referring to... Figure 1 As shown, the method specifically includes:
[0048] The speech keyword recognition model recognizes the following steps: Speech is collected and preprocessed, and initial noise reduction and 160-fold downsampling are performed on high-sampling-rate speech;
[0049] Mel-Cepstral Coefficient Extraction Steps: After initial noise removal and downsampling of high-sampling-rate speech data, Mel-Cepstral coefficients are extracted. Based on traditional coefficient extraction, mixed-window processing and sparse compression processing are adopted, and sinusoidal spectrum enhancement processing is adopted.
[0050] Dynamic time warping calculation steps: After extracting the Mel-frequency cepstral coefficients of the speech data, dynamic time warping calculation preprocessing is required. Based on the traditional warping path calculation, nonlinear difference processing, data alignment processing, normalization processing, and dual threshold decision are adopted.
[0051] After high sampling rate (e.g., 192 kHz) speech data is acquired and input by the speech acquisition device, initial noise cancellation is first performed to remove the first 5% of the speech data; then, 160-fold downsampling processing is performed, which extracts 1 data point from 160 data points.
[0052] After initial noise reduction and downsampling, high-sampling-rate speech data is used to extract Mel-frequency cepstral coefficients. During Mel window construction, partial aliasing is allowed. Based on research on the range of human speech, the highest frequency of the Mel window is set to 2-6 times the range allowed by the sampling theorem, introducing appropriate nonlinearity to improve feature extraction. Considering that the two-dimensional matrix recording the Mel window coefficients contains a large number of zeros, data sparsity is compressed, and the window boundary is recorded to avoid meaningless multiplication by zero. After discrete cosine transform processing, a sine lifting window is added to further improve the accuracy of feature extraction.
[0053] The extracted Mel-Cepstral Coefficients (MCCs) from the speech data require dynamic time warping (VTW) preprocessing. Preprocessing begins with length alignment using nearest-neighbor interpolation, followed by different nonlinear processing on two segments of speech data. Next, frame padding is applied to the processed speech data, and finally, the speech data is divided into blocks of 2-6 frames. The MCCs of the preprocessed segments are then subjected to VTW calculations between corresponding blocks. The results for each block are summed to obtain the overall VTW calculation result. For the overall VTW calculation result, two decision values are calculated: the original decision value and the normalized decision value. The original decision value equals the overall VTW calculation result, and the normalized decision value is obtained by normalizing the overall VTW calculation result for the speech length. Two levels of decision are performed for the two decision values: direct decision and normalized decision. The direct decision determines whether the speech data contains the corresponding keyword by comparing the original decision value with the maximum and minimum thresholds. If the direct decision fails, the normalized decision is performed by comparing the normalized decision value with the degenerate maximum and minimum thresholds to determine whether the speech data contains the corresponding keyword.
[0054] Figure 1The diagram shown is a schematic of a speech keyword recognition system constructed using the present invention. Figure 1 As can be seen, the speech keyword recognition model's recognition structure consists of three parts: a high-sampling-rate speech data preprocessing and downsampling module, a downsampling audio Mel-Cepstral Coefficient extraction module, and a Mel-Cepstral Coefficient dynamic time warping calculation module. When deployed on an edge application, the edge intelligent speech system controller first issues instructions, then the recognition model provided by this invention processes the data and returns the recognition results to the controller. Finally, the intelligent speech controller makes the corresponding system response based on the received recognition results.
[0055] Reference Figure 2 The Mel-Cepstral Coefficient Extraction Process shown in the diagram adds initial noise reduction, 160x downsampling, and sinusoidal lifting window to the traditional coefficient extraction process. It also applies sparsity compression and fast real Fourier transform to significantly save computational and storage resources.
[0056] Turning on the digital microphone at the start of recording introduces significant step noise, rendering this data invalid and impacting the algorithm's effectiveness. Therefore, when receiving audio data, the edge application can skip complex endpoint detection processing and directly discard this data segment. A typical discard length is set to 5% of the total length.
[0057] Even after initial noise cancellation, high-sampling-rate speech data still exhibits significant redundancy. Therefore, high-rate downsampling processing is necessary. After comprehensive consideration, a downsampling rate of 160 is deemed more suitable for processing by edge devices.
[0058] During the Gamel window phase, the frequency response of the Mel triangular filter is defined as shown in equation (1).
[0059]
[0060] In formula (1), That is, H m The sum of the coefficients (k) is 1. Here, m refers to the index of the Mel window, f(m) refers to the discrete mapping of the center frequencies of each Mel window obtained based on the number of Mel windows, the Mel window setting range, and the Mel scale transformation relationship, and k refers to the index of the Mel spectral coefficients, while H... m (k) refers to the value of the kth data point in the m-th Mel window.
[0061] In traditional Mel-frequency coefficient extraction, a two-dimensional array is used to store the Mel-frequency coefficients, which are then multiplied by the speech features obtained from the previous step. As shown in Equation 1, more than half of the parameters in this two-dimensional array are 0, indicating significant data sparsity. By recording the zero data boundaries, the multiplication by zero and subsequent accumulation operations, which have no impact on the results but incur significant computational and storage overhead, can be avoided during Mel window extraction. This can significantly reduce the latency of model computation, which is beneficial for edge devices to achieve real-time keyword recognition.
[0062] In the process of converting Mel spectral coefficients to Mel cepstral coefficients, an important step is the discrete cosine transform. After the traditional discrete cosine transform, a sinusoidal spectrum enhancement is performed, as shown in formula (2):
[0063]
[0064] In formula (2), C0(n) are the coefficients obtained by discrete cosine transform, L is the order of Mel-Cepstral coefficients (typically 12), and C(n) refers to the coefficients obtained by sine boosting.
[0065] Reference Figure 3 The diagram illustrates the Mel-Cepstral Coefficient Extraction process, where partial aliasing is allowed during Mel window construction. Based on research findings on the range of human speech, the highest frequency of the Mel window is set to 2-6 times the range allowed by the sampling theorem, and appropriate nonlinearity is introduced to improve the feature extraction effect.
[0066] In traditional Mel coefficient extraction, the highest frequency of the Mel filter bank can be determined by the Nyquist sampling theorem. Taking a 192kHz high-sampling-rate speech signal as an example, after 160-fold downsampling, the sampling rate is 1200Hz, and the highest frequency without aliasing is only half of that, which is 600Hz. However, a large portion of human voice frequencies exceed 600Hz, and the aforementioned traditional techniques result in insufficient feature extraction, leading to a significant loss in recognition accuracy, indicating room for improvement.
[0067] When identifying keywords, as long as the extracted features are easy to recognize, it is acceptable. Therefore, some aliasing can be tolerated. From another perspective, introducing nonlinear factors may improve the model's performance. Consider selecting a maximum frequency that is 2-6 times the sampling law limit, for example, 4 times 600 Hz, i.e., 2400 Hz. Although this introduces aliasing, it provides a more reasonable consideration of the range of human voices. Figure 3 The reasoning process described above is illustrated.
[0068] Figure 4The diagram shows the preprocessing steps of dynamic time warping calculation in the recognition structure. The Mel-spectral coefficients of two sets of speech data are input into the diagram. First, the speech length is aligned using nearest neighbor interpolation, as shown in formulas (3) and (4).
[0069]
[0070]
[0071] In formulas (3) and (4), A and B are two speech features, and |A| and |B| represent speech lengths. MFCC_ and MFCC_B refer to the Mel-Cepstral coefficients of the two speech segments; BUF_A and BUF_B refer to the coefficients of the two speech segments after preprocessing, and i refers to the index of the coefficients before and after preprocessing.
[0072] Subsequently, nonlinear difference processing is performed on the Mel-Cepstral coefficient data of the aligned speech, as shown in Equations (5) and (6).
[0073]
[0074]
[0075] In formulas (5) and (6), the meanings of A and B, |A| and |B|, and BUF_A and BUF_B are the same as in formulas (3) and (4). DIFF_A and DIFF_B refer to the coefficients after nonlinear differential processing of two speech segments, while i refers to the index of the coefficients before and after nonlinear differential processing.
[0076] Finally, the Mel-Cepstral Coefficients (MCCs) data of the differentially processed speech are padded with frames. This is done by adding a subtracted value of -1 to the data, ensuring the total MCC data size is a multiple of k frames in the speech. This facilitates block processing. After padded frames, the data is then divided into blocks of k frames each. Here, k is the block parameter, typically 3.
[0077] Reference Figure 5 The diagram shows the dynamic time warping calculation process in the dynamic time warping calculation module of the recognition structure. The two preprocessed segments of speech feature data are input into the diagram. A dynamic programming algorithm is used to calculate the shortest warped path distance between corresponding segments. The results are then summed to obtain the overall dynamic time warping calculation result.
[0078] Reference Figure 6The diagram shows the post-processing procedure of dynamic time warping calculation in the dynamic time warping calculation module of the recognition structure. The overall dynamic time warping calculation result input in the figure requires the calculation of two decision values. The original decision value is consistent with the overall dynamic time warping calculation result, and the normalized decision value is obtained by normalizing the speech length information in the overall dynamic time warping calculation result. The specific normalization method is shown in formula (7) and formula (8).
[0079]
[0080]
[0081] In Formula 7, the meanings of |A| and |B| are the same as in Formulas (3) and (4), while ratio refers to the ratio of the lengths of the two speech features.
[0082] In formula (8), VALUE is the overall dynamic time warping calculation result, ratiio has the same meaning as in formula (7), LIMIT refers to the threshold for speech length normalization, and VALUE_NORM refers to the decision value after normalization. A typical value for LIMIT is 1.2.
[0083] Reference Figure 7 The diagram shows the dynamic time warping calculation decision process in the dynamic time warping calculation module of the recognition structure. The input original decision value and normalized decision value in the figure need to be used for two-level decision. The direct decision determines whether the speech data contains the corresponding keyword by comparing the relationship between the original decision value and the maximum and minimum thresholds. The specific decision method is shown in formula (9).
[0084] VALUE_MIN≤VALUE_DTW≤VALUE_MAX (9)
[0085] In formula (9), VALUE_DTW refers to the dynamic time warping decision value, while VALUE_MIN and VALUE_MAX refer to the maximum and minimum thresholds of the direct decision obtained by model training, respectively.
[0086] If the direct judgment fails, a normalization judgment is performed. The normalization judgment determines whether the speech data contains the corresponding keyword by comparing the relationship between the normalization judgment value and the degenerate maximum and minimum thresholds. The specific judgment method is shown in formula (10).
[0087] VALUE_MIN-NORM_PARA≤VALUE_NORM≤VALUE_MAX-NORM_PARA (10)
[0088] In formula (10), VALUE_NORM has the same meaning as in formula (8), VALUE_MIN and VALUE_MAX have the same meaning as in formula (9), and NORM_PARA refers to the normalization adjustment factor obtained from the model training for the second-level normalization decision, with a typical value of 5. The corresponding adjusted thresholds are the degenerate maximum and minimum thresholds used in the normalization decision.
[0089] This invention also provides a hardware-compatible speech keyword recognition system. This hardware-compatible speech keyword recognition system can be implemented by executing the steps of the hardware-compatible speech keyword recognition method. That is, those skilled in the art can understand the hardware-compatible speech keyword recognition method as a preferred embodiment of the hardware-compatible speech keyword recognition system. Specifically, the system includes:
[0090] Speech Keyword Recognition Model Recognition Module: Collects speech and performs speech preprocessing, including initial noise reduction and 160x downsampling for high-sampling-rate speech;
[0091] Mel-Cepstral Coefficient Extraction Module: After initial noise removal and downsampling of high-sampling-rate speech data, Mel-Cepstral coefficients are extracted. Based on traditional coefficient extraction, mixed-window processing and sparse compression processing are adopted, and sinusoidal spectrum enhancement processing is used.
[0092] Dynamic time warping calculation module: After extracting the Mel-frequency cepstral coefficients of the speech data, dynamic time warping calculation preprocessing is required. Based on the traditional warping path calculation, nonlinear difference processing, data alignment processing, normalization processing, and dual threshold decision are adopted.
[0093] After high sampling rate (e.g., 192 kHz) speech data is acquired and input by the speech acquisition device, initial noise cancellation is first performed to remove the first 5% of the speech data; then, 160-fold downsampling processing is performed, which extracts 1 data point from 160 data points.
[0094] After initial noise reduction and downsampling, high-sampling-rate speech data is used to extract Mel-frequency cepstral coefficients. During Mel window construction, partial aliasing is allowed. Based on research on the range of human speech, the highest frequency of the Mel window is set to 2-6 times the range allowed by the sampling theorem, introducing appropriate nonlinearity to improve feature extraction. Considering that the two-dimensional matrix recording the Mel window coefficients contains a large number of zeros, data sparsity is compressed, and the window boundary is recorded to avoid meaningless multiplication by zero. After discrete cosine transform processing, a sine lifting window is added to further improve the accuracy of feature extraction.
[0095] The extracted Mel-Cepstral Coefficients (MCCs) from the speech data require dynamic time warping (VTW) preprocessing. Preprocessing begins with length alignment using nearest-neighbor interpolation, followed by different nonlinear processing on two segments of speech data. Next, frame padding is applied to the processed speech data, and finally, the speech data is divided into blocks of 2-6 frames. The MCCs of the preprocessed segments are then subjected to VTW calculations between corresponding blocks. The results for each block are summed to obtain the overall VTW calculation result. For the overall VTW calculation result, two decision values are calculated: the original decision value and the normalized decision value. The original decision value equals the overall VTW calculation result, and the normalized decision value is obtained by normalizing the overall VTW calculation result for the speech length. Two levels of decision are performed for the two decision values: direct decision and normalized decision. The direct decision determines whether the speech data contains the corresponding keyword by comparing the original decision value with the maximum and minimum thresholds. If the direct decision fails, the normalized decision is performed by comparing the normalized decision value with the degenerate maximum and minimum thresholds to determine whether the speech data contains the corresponding keyword.
[0096] Figure 1 The diagram shown is a schematic of a speech keyword recognition system constructed using the present invention. Figure 1 As can be seen, the speech keyword recognition model's recognition structure consists of three parts: a high-sampling-rate speech data preprocessing and downsampling module, a downsampling audio Mel-Cepstral Coefficient extraction module, and a Mel-Cepstral Coefficient dynamic time warping calculation module. When deployed on an edge application, the edge intelligent speech system controller first issues instructions, then the recognition model provided by this invention processes the data and returns the recognition results to the controller. Finally, the intelligent speech controller makes the corresponding system response based on the received recognition results.
[0097] Reference Figure 2 The Mel-Cepstral Coefficient Extraction Process shown in the diagram adds initial noise reduction, 160x downsampling, and sinusoidal lifting window to the traditional coefficient extraction process. It also applies sparsity compression and fast real Fourier transform to significantly save computational and storage resources.
[0098] Turning on the digital microphone at the start of recording introduces significant step noise, rendering this data invalid and impacting the algorithm's effectiveness. Therefore, when receiving audio data, the edge application can skip complex endpoint detection processing and directly discard this data segment. A typical discard length is set to 5% of the total length.
[0099] Even after initial noise cancellation, high-sampling-rate speech data still exhibits significant redundancy. Therefore, high-rate downsampling processing is necessary. After comprehensive consideration, a downsampling rate of 160 is deemed more suitable for processing by edge devices.
[0100] During the Gamel window phase, the frequency response of the Mel triangular filter is defined as shown in equation (1).
[0101]
[0102] In formula (1), That is, H m The sum of the coefficients (k) is 1. Here, m refers to the index of the Mel window, f(m) refers to the discrete mapping of the center frequencies of each Mel window obtained based on the number of Mel windows, the Mel window setting range, and the Mel scale transformation relationship, k refers to the index of the Mel spectral coefficients, and H... m (k) refers to the value of the kth data point in the m-th Mel window.
[0103] In traditional Mel-frequency coefficient extraction, a two-dimensional array is used to store the Mel-frequency coefficients, which are then multiplied by the speech features obtained from the previous step. As shown in Equation 1, more than half of the parameters in this two-dimensional array are 0, indicating significant data sparsity. By recording the zero data boundaries, the multiplication by zero and subsequent accumulation operations, which have no impact on the results but incur significant computational and storage overhead, can be avoided during Mel window extraction. This can significantly reduce the latency of model computation, which is beneficial for edge devices to achieve real-time keyword recognition.
[0104] In the process of converting Mel spectral coefficients to Mel cepstral coefficients, an important step is the discrete cosine transform. After the traditional discrete cosine transform, a sinusoidal spectrum enhancement is performed, as shown in formula (2):
[0105]
[0106] In formula (2), C0(n) are the coefficients obtained by discrete cosine transform, L is the order of Mel-Cepstral coefficients (typically 12), and C(n) refers to the coefficients obtained by sine boosting.
[0107] Reference Figure 3 The diagram illustrates the Mel-Cepstral Coefficient Extraction process, where partial aliasing is allowed during Mel window construction. Based on research findings on the range of human speech, the highest frequency of the Mel window is set to 2-6 times the range allowed by the sampling theorem, and appropriate nonlinearity is introduced to improve the feature extraction effect.
[0108] In traditional Mel coefficient extraction, the highest frequency of the Mel filter bank can be determined by the Nyquist sampling theorem. Taking a 192kHz high-sampling-rate speech signal as an example, after 160-fold downsampling, the sampling rate is 1200Hz, and the highest frequency without aliasing is only half of that, which is 600Hz. However, a large portion of human voice frequencies exceed 600Hz, and the aforementioned traditional techniques result in insufficient feature extraction, leading to a significant loss in recognition accuracy, indicating room for improvement.
[0109] When identifying keywords, as long as the extracted features are easy to recognize, it is acceptable. Therefore, some aliasing can be tolerated. From another perspective, introducing nonlinear factors may improve the model's performance. Consider selecting a maximum frequency that is 2-6 times the sampling law limit, for example, 4 times 600 Hz, i.e., 2400 Hz. Although this introduces aliasing, it provides a more reasonable consideration of the range of human voices. Figure 3 The reasoning process described above is illustrated.
[0110] Figure 4 The diagram shows the preprocessing steps of dynamic time warping calculation in the recognition structure. The Mel-spectral coefficients of two sets of speech data are input into the diagram. First, the speech length is aligned using nearest neighbor interpolation, as shown in formulas (3) and (4).
[0111]
[0112]
[0113] In formulas (3) and (4), A and B are two speech features, and |A| and |B| represent speech lengths. MFCC_A and MFCC_B refer to the Mel-Cepstral coefficients of the two speech segments; BUF_A and BUF_B refer to the coefficients of the two speech segments after preprocessing, and i refers to the index of the coefficients before and after preprocessing.
[0114] Subsequently, nonlinear difference processing is performed on the Mel-Cepstral coefficient data of the aligned speech, as shown in Equations (5) and (6).
[0115]
[0116]
[0117] In formulas (5) and (6), the meanings of A and B, |A| and |B|, and BUF_A and BUF_B are the same as in formulas (3) and (4). DIFF_A and DIFF_B refer to the coefficients after nonlinear differential processing of two speech segments, while i refers to the index of the coefficients before and after nonlinear differential processing.
[0118] Finally, the Mel-Cepstral Coefficients (MCCs) data of the differentially processed speech are padded with frames. This is done by adding a subtracted value of -1 to the data, ensuring the total MCC data size is a multiple of k frames in the speech. This facilitates block processing. After padded frames, the data is then divided into blocks of k frames each. Here, k is the block parameter, typically 3.
[0119] Reference Figure 5The diagram shows the dynamic time warping calculation process in the dynamic time warping calculation module of the recognition structure. The two preprocessed segments of speech feature data are input into the diagram. A dynamic programming algorithm is used to calculate the shortest warped path distance between corresponding segments. The results are then summed to obtain the overall dynamic time warping calculation result.
[0120] Reference Figure 6 The diagram shows the post-processing procedure of dynamic time warping calculation in the dynamic time warping calculation module of the recognition structure. The overall dynamic time warping calculation result input in the figure requires the calculation of two decision values. The original decision value is consistent with the overall dynamic time warping calculation result, and the normalized decision value is obtained by normalizing the speech length information in the overall dynamic time warping calculation result. The specific normalization method is shown in formula (7) and formula (8).
[0121]
[0122]
[0123] In Formula 7, the meanings of |A| and |B| are the same as in Formulas (3) and (4), while ratio refers to the ratio of the lengths of the two speech features.
[0124] In formula (8), VALUE is the overall dynamic time warping calculation result, ratio has the same meaning as in formula (7), LIMIT refers to the threshold for speech length normalization, and VALUE_NORM refers to the decision value after normalization. A typical value for LIMIT is 1.2.
[0125] Reference Figure 7 The diagram shows the dynamic time warping calculation decision process in the dynamic time warping calculation module of the recognition structure. The input original decision value and normalized decision value in the figure need to be used for two-level decision. The direct decision determines whether the speech data contains the corresponding keyword by comparing the relationship between the original decision value and the maximum and minimum thresholds. The specific decision method is shown in formula (9).
[0126] VALUE_MIN≤VALUE_DTW≤VALUE_MAX (9)
[0127] In formula (9), VALUE_DTW refers to the dynamic time warping decision value, while VALUE_MIN and VALUE_MAX refer to the maximum and minimum thresholds of the direct decision obtained by model training, respectively.
[0128] If the direct judgment fails, a normalization judgment is performed. The normalization judgment determines whether the speech data contains the corresponding keyword by comparing the relationship between the normalization judgment value and the degenerate maximum and minimum thresholds. The specific judgment method is shown in formula (10).
[0129] VALUE_MIN-NORM_PARA≤VALUE_NORM≤VALUE_MAX-NORM_PARA (10)
[0130] In formula (10), VALUE_NORM has the same meaning as in formula (8), VALUE_MIN and VALUE_MAX have the same meaning as in formula (9), and NORM_PARA refers to the normalization adjustment factor obtained from the model training for the second-level normalization decision, with a typical value of 5. The corresponding adjusted thresholds are the degenerate maximum and minimum thresholds used in the normalization decision.
[0131] This invention provides a speech keyword recognition method and system suitable for hardware processing, which can effectively reduce hardware computing and storage overhead, realize real-time speech keyword recognition processing, be deployed and processed on edge hardware, and ensure a high speech keyword recognition accuracy.
[0132] Those skilled in the art will understand that, besides implementing the system and its various devices, modules, and units provided by this invention in the form of purely computer-readable program code, the same functions can be achieved entirely through logical programming of the method steps, making the system and its various devices, modules, and units of this invention function in the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers. Therefore, the system and its various devices, modules, and units provided by this invention can be considered as a hardware component, and the devices, modules, and units included therein for implementing various functions can also be considered as structures within the hardware component; alternatively, the devices, modules, and units for implementing various functions can be considered as both software modules implementing the method and structures within the hardware component.
[0133] Specific embodiments of the present invention have been described above. It should be understood that the present invention is not limited to the specific embodiments described above, and those skilled in the art can make various changes or modifications within the scope of the claims, which do not affect the essence of the present invention. Unless otherwise specified, the embodiments and features described in this application can be arbitrarily combined with each other.
Claims
1. A speech keyword recognition method suitable for hardware processing, characterized in that, include: The speech keyword recognition model recognizes the following steps: Speech is collected and preprocessed, and initial noise reduction and downsampling are performed on high-sampling-rate speech; Mel-Cepstral Coefficient Extraction Steps: After initial noise removal and downsampling of high-sampling-rate speech data, Mel-Cepstral coefficients are extracted. Based on the coefficient extraction, mixed-window processing and sparse compression processing are adopted, and sinusoidal spectrum enhancement processing is adopted. Dynamic time warping calculation steps: After extracting the Mel-frequency cepstral coefficients of the speech data, dynamic time warping calculation preprocessing is required. Based on the warping path calculation, nonlinear difference processing, data alignment processing, normalization processing, and dual threshold decision are adopted. In the Mel cepstral coefficient extraction step, Mel cepstral coefficients are extracted. During the Mel window construction process, partial aliasing is allowed, and the highest frequency of the Mel window is set to 2-6 times that allowed by the sampling theorem.
2. The speech keyword recognition method suitable for hardware processing according to claim 1, characterized in that, In the speech keyword recognition model recognition step, after the high sampling rate speech data is collected and input by the speech acquisition device, initial noise cancellation is first performed to remove the first 5% of the speech data; then, 160-fold downsampling processing is performed to extract 1 data from 160 data points.
3. The speech keyword recognition method suitable for hardware processing according to claim 1, characterized in that, During the construction of the Mel window, data sparsity is compressed, and the window boundary is recorded to avoid multiplication by zero; after discrete cosine transform processing, a sine lifting window is added.
4. The speech keyword recognition method suitable for hardware processing according to claim 1, characterized in that, The dynamic time warping calculation steps include: extracting the Mel-Cepstral Coefficients of the speech data, which need to be preprocessed by dynamic time warping calculation. The preprocessing first uses nearest neighbor interpolation to align the length of the speech data, then performs different nonlinear processing on the two segments of speech data, then performs frame padding on the processed speech data, and finally divides the speech data into blocks of 2-6 frames each.
5. The speech keyword recognition method suitable for hardware processing according to claim 4, characterized in that, The speech Mel-spectral coefficients obtained after dynamic time warping calculation preprocessing are divided into blocks. Dynamic time warping calculation is then performed between corresponding blocks. The results of the calculation for each block are summed to obtain the overall dynamic time warping calculation result.
6. The speech keyword recognition method suitable for hardware processing according to claim 5, characterized in that, For the overall dynamic time warping calculation results, two decision values need to be calculated: the original decision value and the normalized decision value; The original decision value is equal to the result of the overall dynamic time warping calculation. The normalized decision value is obtained by normalizing the speech length based on the result of the overall dynamic time warping calculation.
7. The speech keyword recognition method suitable for hardware processing according to claim 6, characterized in that, Two levels of decision are made for the two decision values: direct decision and normalized decision; the direct decision determines whether the speech data contains the corresponding keyword by comparing the original decision value with the maximum and minimum thresholds. If the direct judgment fails, a normalization judgment is performed. The normalization judgment determines whether the speech data contains the corresponding keyword by comparing the relationship between the normalization judgment value and the degenerate maximum and minimum thresholds.
8. A voice keyword recognition system suitable for hardware processing, characterized in that, include: Speech Keyword Recognition Model Recognition Module: Collects speech and performs speech preprocessing, including initial noise reduction and downsampling for high-sampling-rate speech; Mel-Cepstral Coefficient Extraction Module: After initial noise removal and downsampling of high-sampling-rate speech data, Mel-Cepstral coefficients are extracted. Based on the coefficient extraction, mixed-window processing and sparse compression processing are adopted, and sinusoidal spectrum enhancement processing is adopted. Dynamic time warping calculation module: After extracting the Mel-frequency cepstral coefficients of the speech data, dynamic time warping calculation preprocessing is required. Based on the warping path calculation, nonlinear difference processing, data alignment processing, normalization processing and dual threshold decision are adopted. In the Mel cepstral coefficient extraction module, Mel cepstral coefficients are extracted. During the Mel window construction process, partial aliasing is allowed, and the highest frequency of the Mel window is set to 2-6 times that allowed by the sampling theorem.
9. The speech keyword recognition system suitable for hardware processing according to claim 8, characterized in that, In the speech keyword recognition model recognition module, after the high sampling rate speech data is collected and input by the speech acquisition device, it first performs initial noise cancellation to remove the first 5% of the speech data; then it performs 160x downsampling processing to extract 1 data from 160 data. During the construction of the Mel window, data sparsity is compressed, and the window boundary is recorded to avoid multiplication by zero; after discrete cosine transform processing, a sine lifting window is added.
10. The speech keyword recognition system suitable for hardware processing according to claim 8, characterized in that, The dynamic time warping calculation module includes: extracting Mel-frequency cepstral coefficients from the speech data, which requires dynamic time warping calculation preprocessing. The preprocessing first uses nearest neighbor interpolation to align the length of the speech data, then performs different nonlinear processing on the two segments of speech data, then performs frame padding on the processed speech data, and finally divides the speech data into blocks of 2-6 frames. The speech Mel-spectral coefficients of the blocks obtained after dynamic time warping calculation preprocessing are then subjected to dynamic time warping calculation between corresponding blocks. The results of the calculation for each block are then summed to obtain the overall dynamic time warping calculation result. For the overall dynamic time warping calculation results, two decision values need to be calculated: the original decision value and the normalized decision value; The original decision value is equal to the overall dynamic time warping calculation result. The normalized decision value is obtained by normalizing the speech length based on the overall dynamic time warping calculation result. Two levels of decision are performed for the two decision values: direct decision and normalized decision. The direct decision determines whether the speech data contains the corresponding keyword by comparing the original decision value with the maximum and minimum thresholds. If the direct decision fails, the normalized decision is performed by comparing the normalized decision value with the degenerate maximum and minimum thresholds to determine whether the speech data contains the corresponding keyword.