An Efficient Audio-to-Spike Pipeline for Spiking Neural Networks

The audio-spike conversion pipeline optimizes neuromorphic processors for edge devices, addressing computing power and bandwidth limitations in real-time speech recognition, enhancing accuracy and efficiency.

JP7583088B2Active Publication Date: 2024-11-13ACCENTURE GLOBAL SERVICES LTD
View PDF 8 Cites 0 Cited by

Patent Information

Application Number
JP2023035501
Authority / Receiving Office
JP · JP
Patent Type
Patents
Current Assignee / Owner
Priority Date
2023-03-07
Filing Date
2023-03-08
Publication Date
2024-11-13
Estimated Expiration
2043-03-08

AI Technical Summary

Technical Problem

Existing edge devices face limitations in computing power and bandwidth when performing real-time speech recognition, necessitating more efficient audio processing techniques to enhance performance.

Method used

An audio-spike conversion pipeline utilizing a neuromorphic processor and spiking neural networks (SNNs) for edge devices, optimized through hyperparameter tuning to convert audio data into spikes efficiently, reducing computational overhead and latency.

Benefits of technology

Enables real-time speech recognition on power-constrained devices with improved accuracy and reduced bandwidth consumption, facilitating widespread adoption of neuromorphic accelerators in edge devices for voice control and continuous learning.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 0007583088000003
    Figure 0007583088000003
  • Figure 0007583088000004
    Figure 0007583088000004
  • Figure 0007583088000005
    Figure 0007583088000005
Patent Text Reader

Abstract

To provide a method, a system, and an apparatus including a computer program encoded on a computer storage medium, for converting audio to a spike for input to a spiking neural network configured to recognize a speech based on the spike.SOLUTION: A method includes the steps of: obtaining audio data; generating a frequency domain audio signal that represents the audio data by converting the audio data into a frequency domain; mapping the frequency domain audio signal into a set of Mel-frequency bands and obtaining a log-Mel signal; converting a log transformation on a Mel-scale frequency audio signal to an execution spike; and generating a spike input to be input to a spiking neural network (SNN) model by converting the log-Mel signal to a series of spikes.SELECTED DRAWING: Figure 3
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] This specification relates generally to audio processing and machine learning.

[0002] CROSS-REFERENCE TO RELATED APPLICATIONS This application claims priority to U.S. Provisional Patent Application No. 63 / 317,873, filed March 8, 2022, and U.S. Patent Application No. 18 / 118,619, filed March 7, 2023, the entire contents of which are incorporated by reference herein for all purposes. [Background technology]

[0003] Speech recognition involves the technology of recognizing spoken language and converting it into text. Speech recognition is used in many contexts, including speech-to-text processes and voice interfaces where people interact with electronic devices using voice commands. Summary of the Invention [Means for solving the problem]

[0004] Edge computing is a distributed computing paradigm that brings computation and data storage closer to the source of data. Edge computing can improve response times and save bandwidth. Edge devices are computing devices that process data at a local level. However, the computing power of edge devices may be limited compared to traditional server computers.

[0005] Brain-inspired neuromorphic hardware and spiking neural network models (SNNs) running on neuromorphic hardware can offer compelling advantages that make them suitable for speech recognition for edge devices. Neuromorphic processors have the advantages of being low power, low latency, and continuously learning. As an example, neuromorphic processors may reduce the power consumed during speech recognition inference by several orders of magnitude compared to CPUs or GPU accelerators. Neuromorphic processors can enable a variety of power-constrained devices, such as edge devices, to support speech recognition using a larger vocabulary than other speech recognition techniques, such as digital signal processing. Neuromorphic systems process sparse, event-based data that can enable low-latency inference across diverse applications, including speech recognition. Neuromorphic systems can support continuous learning algorithms for SNNs.

[0006] The techniques described herein provide an efficient audio-to-spike conversion pipeline that converts speech into spikes that are input to an SNN, and that is suitable for real-time operation using low-power neuromorphic processors so that edge devices can perform the conversion and speech recognition quickly and efficiently. By way of example, this document describes a pipeline and corresponding techniques for converting raw audio, including streamed audio, into spikes fast enough for real-time speech recognition on edge devices.

[0007] The audio-to-spike conversion pipeline may include hyperparameters that control how the pipeline processes the audio data to convert the audio data into spike inputs for the SNN. A hyperparameter sweep may be performed on the audio-spike pipeline to optimize or at least improve the hyperparameters of the pipeline, resulting in a more efficient (e.g., requiring less computation and / or fewer processor cycles) and faster conversion of the audio data into spikes. By way of example, the hyperparameter sweep process may determine the relative importance of hyperparameters, such that processing associated with some hyperparameters may no longer be performed if the hyperparameters are not important. This reduces the amount of processing of the audio data (or intermediate forms of the audio data) without adversely affecting the accuracy of the conversion.

[0008] According to some implementations, a method performed by one or more data processing devices includes obtaining audio data, generating a frequency domain audio signal representing the audio data by transforming the audio data into the frequency domain, mapping the frequency domain audio signal to a set of Mel frequency bands to obtain a Mel scale frequency audio signal, performing a logarithmic transformation on the Mel scale frequency audio signal to obtain a logarithmic Mel signal, generating a spike input comprising a series of spikes input to a Spiking Neural Network (SNN) model by transforming the logarithmic Mel signal into a series of spikes, and providing the spike input as an input to the SNN model. Other embodiments of this aspect include corresponding computer systems, devices, computer program products, and computer programs recorded on one or more computer storage devices, each configured to perform the actions of the method. One or more computer systems can be configured to perform a particular operation or action by having software, firmware, hardware, or a combination thereof installed on the system that causes the system to perform an action during operation. One or more computer programs can be configured to perform a particular operation or action by including instructions that, when executed by a data processing device, cause the device to perform an action.

[0009] Each implementation may include one or more of the following features. In some aspects, the method includes running an SNN model using the spike input. The spiking neural network processes the series of spikes to generate a speech recognition result including text corresponding to the audio data, and outputs the speech recognition result.

[0010] Some aspects include performing an action based on the speech recognition result. In some aspects, obtaining the audio data includes obtaining the audio data from a memory or obtaining the audio data from an active microphone.

[0011] In some aspects, transforming the audio data to the frequency domain includes centering and scaling the audio signal to obtain centered and scaled audio data, and transforming the centered and scaled audio data to the frequency domain using a sliding window of a predetermined length. In some aspects, transforming the audio data to the frequency domain includes transforming the audio data using a sliding discrete Fourier transform (SDFT) algorithm.

[0012] In some aspects, converting the log-Mel signals into spike inputs includes determining a step-by-step difference between the log-Mel signals of two adjacent time steps, determining that the step-by-step difference meets a threshold in either a positive or negative direction, and generating a spike of the corresponding polarity. In some aspects, converting the log-Mel signals into spikes includes determining a running sum spike by calculating a quantized sum of each log-Mel signal.

[0013] In some aspects, converting the log-mel features into spikes includes stacking the log-mel features multiple times to obtain features stacked in multiple blocks, and converting the features stacked in multiple blocks into spikes.

[0014] In some aspects, one or more components of the audio spike transform pipeline include hyperparameters that are adjusted based on classification accuracy. Some aspects include, for each hyperparameter setting in a plurality of sets of hyperparameter settings, applying the hyperparameter setting to one or more components of the audio spike transform pipeline, training a plurality of test SNN models using training audio data that is converted to spike inputs using one or more components of the audio spike transform pipeline, testing each test SNN model using the test data, including generating result data by determining accuracy of speech recognition results produced by each test SNN model, and updating the hyperparameters based on the result data for each set of hyperparameters.

[0015] Some aspects include, for each set of hyperparameters, determining an importance of each hyperparameter based on the outcome data, and updating the hyperparameters includes updating at least one hyperparameter based on the importance of the at least one hyperparameter.

[0016] Some aspects include avoiding the use of certain hyper-parameters when generating spike inputs based on the importance of a given hyper-parameter.

[0017] Methods according to the present disclosure may include any combination of the aspects and features described herein, i.e., methods according to the present disclosure are not limited to the combinations of the aspects and features specifically described herein, but may include any combination of the aspects and features shown.

[0018] The details of one or more implementations of the disclosure are set forth in the accompanying drawings and the description below. Other features and advantages of the disclosure will become apparent from the description and drawings, and from the claims. [Brief description of the drawings]

[0019] [Figure 1] FIG. 1 is a block diagram of an example environment in which a computing device processes audio data to recognize speech sounds within the audio data. [Diagram 2] An example shows the four stages of audio-spike conversion. [Diagram 3] FIG. 1 is a flow diagram of an example process for converting audio data into spikes and processing the spikes using an SNN to recognize speech sounds. [Figure 4A] 4 illustrates an example process for generating spikes in an audio signal. [Figure 4B] 4 illustrates an example process for generating spikes in an audio signal. [Diagram 5] FIG. 13 is a flow diagram of an example process for tuning hyper-parameters of an audio-spike transform pipeline. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0020] The same reference numbers and designations in different drawings indicate the same elements.

[0021] The present specification generally relates to an audio-to-spike conversion pipeline that can encode audio data into spikes that are input to an SNN, which is trained to generate speech recognition results that include predicted speech sounds in the audio data. The audio-to-spike conversion pipeline includes a set of computational operations, each of which is configured, e.g., optimized, for efficient real-time execution and to generate an information-rich, sparse representation of the raw audio signal. The pipeline can include a combination of stages that provide low latency and require minimal computational overhead, and converts audio data, which can be streaming audio input received by the pipeline, into spikes that can be used for real-time speech recognition by power-constrained devices, such as edge devices with low-power neuromorphic processors.

[0022] 1 is a block diagram of an example environment 100 in which a computing device 102 processes audio data to recognize speech sounds within the audio data. The example environment 100 includes a computing device 102, a remote server 110, a cloud server 112, and a data communication network 104, such as a local area network (LAN), a wide area network (WAN), the Internet, a mobile network, or a combination thereof. Note that, as described below, all of the audio processing can occur at the computing device 102 without sending or receiving data from any of the servers 110, 112, which increases the speed at which speech is recognized at the computing device 102 and reduces bandwidth consumption on the network 104, for example, by eliminating round-trip communication over the network 104.

[0023] The computing device 102 may obtain the audio data from a microphone 106 or a storage device 108. In some examples, the microphone 106 may be a component (e.g., a sensor) included in the computing device 102. In other examples, the microphone 106 may be an Internet of Things (IoT) device, a networked device, or a wired device connected to the computing device 102. In some examples, the computing device 102 may obtain the audio data from a memory, such as the memory of the computing device 102, or a data storage device 108. In either case, the audio data may be in the form of raw (e.g., unprocessed) audio and may include speech in the form of spoken words.

[0024] In some implementations, the computing device 102 may be an edge device that processes data at a local level. As an example, an Internet of Things (IoT) device may generate a large amount of data during its operation. The edge device may process IoT data locally to reduce the bandwidth needs of the large amount of data, instead of transmitting the IoT data over the network 104 to the remote server 110 or the cloud server 112. In some implementations, the edge device may be a device that is power constrained compared to the remote server 110 and the cloud server 112. The edge device may be an IoT computing device, a smart camera, a smart speaker, an embedded computer, a mobile device, or even a smart TV or other connected device. In a particular example, the computing device 102 may be a smart speaker or a smartphone that includes and / or is communicatively coupled (e.g., via the network 104) to an artificial intelligence (AI) assistant that performs tasks based on the recognized voice of a user.

[0025] The computing device 102 includes an audio-to-spike transform pipeline 122 and a neuromorphic processor 124. The audio-to-spike transform pipeline 122 may be implemented using hardware (e.g., a processor such as a central processing unit (CPU)) and / or software.

[0026] The audio spike conversion pipeline 122 is configured to convert audio data into spikes for processing by the neuromorphic processor 124. The series of spikes output by the audio spike conversion pipeline 122 for the audio data can be referred to as speech input, which is provided as an input to the SNN 126 executed by the neuromorphic processor 124. The audio spike conversion pipeline 122 can include multiple processing stages for converting the audio data into spikes. These stages are described in more detail with reference to Figures 2-4B. The audio spike conversion pipeline 122 is efficient and suitable for real-time operation. By way of example, the stages of the audio spike conversion pipeline 122 are configured to convert streamed raw audio into spikes fast enough for real-time speech recognition by the neuromorphic processor 124.

[0027] The neuromorphic processor 124 is configured to perform speech recognition tasks using the SNN 126. As an example, the neuromorphic processor 124 may execute the SNN 126 to predict speech sounds represented by spikes in a spike input output by the audio-to-spike transform pipeline 122. As an example, the SNN 126 may process the spikes to generate speech recognition results that include predicted speech sounds corresponding to the sounds of the audio data, e.g., in the form of phonemes, triphones, whole words, or other sound units of text.

[0028] In some implementations, for example when the SNN 126 is not configured to output words as the predicted speech sounds, the computing device 102 may include a speech recognizer 130 configured to determine words and / or sentences that correspond to the phonemes or other non-word predicted speech sounds output by the neuromorphic processor 124. By way of example, the speech recognizer 130 may include a lexicon and / or a language model configured to convert phonemes into words and / or sentences. In this manner, the computing device 102 may convert audio data into words, phrases, and / or sentences without requiring network communication.

[0029] In some implementations, the computing device 102 can send the predicted speech sounds to the server(s) 110, 112 for conversion of the predicted speech sounds into words and / or sentences. By way of example, one or both of the servers 110, 112 can include a speech recognizer 130 or can include multiple speech recognizer components, such as, for example, separate language models for different regions and their various languages ​​and dialects. The server(s) 110, 112 can return the words and / or sentences to the computing device 102 or perform actions based on the words and / or sentences. By way of example, the server(s) 110, 112 can include an AI assistant that performs tasks based on voice commands received from the user's computing device 102.

[0030] One or both of the servers 110, 112 may include a hyperparameter tuner 128 that may be implemented in hardware and / or software. In some implementations, the hyperparameter tuner 128 may be part of the computing device 102. The hyperparameter tuner 128 may evaluate the SNNs and tune the hyperparameters of the audio spike transform pipeline 122 based on the evaluation. Exemplary hyperparameters of the audio spike transform pipeline 122 are described with reference to FIG. 2. As an example, the hyperparameter tuner 128 may perform a hyperparameter sweep using multiple sets of hyperparameter settings and select a hyperparameter setting based on the results of the sweep. Each sweep may include applying a particular hyperparameter setting to the audio spike transform pipeline 122, training multiple SNNs using training data and spikes generated by the audio spike transform pipeline 122 based on the training data, testing each of the multiple SNNs using test data, and measuring the accuracy of each SNN based on the testing. The training data and / or test data may include labeled audio data with labels indicating predicted speech sounds, e.g., words or phonemes, corresponding to the audio data.

[0031] Based on the evaluation, hyperparameter tuner 128 can tune hyperparameters of audio-to-spike transform pipeline 122. As an example, hyperparameter tuner 128 can identify the hyperparameter setting that trained the most accurate SNN during a sweep and apply the hyperparameter setting to audio-to-spike transform pipeline 122.

[0032] Hyperparameter tuner 128 may also determine the relative importance of each hyperparameter based on the results of each sweep. By way of example, if there is a significant change in the accuracy of the SNN trained during a sweep when the setting of a particular hyperparameter is adjusted, then that particular hyperparameter is likely to have a significant impact on the accuracy of the SNN and is therefore important. In assessing the importance of each hyperparameter, hyperparameter tuner 128 may evaluate various factors, such as the change in accuracy of the SNN with changes to the hyperparameter, whether the hyperparameter contributed to the most accurate SNN or at least a threshold number of the most accurate SNNs, the computational cost of the hyperparameter, the power required by the hyperparameter, minimization of a loss function, and / or other suitable factors.

[0033] Based on the relative importance of the hyper-parameters, the hyper-parameter tuner 128 can make changes to the audio-to-spike transform pipeline 122. As an example, the hyper-parameter tuner 128 can determine an updated configuration for the audio-to-spike transform pipeline 122 (e.g., updated code for software components) and send the updated configuration to the computing device 102.

[0034] In some cases, the hyperparameter tuner 128 may modify the settings of hyperparameters corresponding to certain software components of the audio-to-spike transform pipeline 122 such that the software components are no longer active if the hyperparameter is determined to be of no or low importance. Processing the audio data or intermediate forms of the audio data based on each hyperparameter has an associated computational cost (e.g., CPU cycles) and latency. By removing those of lesser importance, the conversion of the audio data to spikes can be performed faster and more efficiently with little or no adverse effect on the accuracy of the speech recognition results of the SNN 126. It should be noted that some hyperparameters may be more important for some languages ​​and / or dialects. Thus, the ability to use a large number of hyperparameters and filter some based on their importance allows for highly accurate speech recognition with a highly efficient speech-to-spike transform.

[0035] In some implementations, the hyperparameter tuner 128 selects which hyperparameters to use or not use based on the relative importance and / or computational cost of each hyperparameter. As an example, if a hyperparameter is of medium importance but has the highest computational cost, the hyperparameter tuner 128 may turn it off to increase the efficiency of the conversion to spikes. In some implementations, a user, e.g., a software developer, may make these selections and configure the audio-to-spike transform pipeline 122 accordingly. An example process for tuning the hyperparameters of the audio-to-spike transform pipeline is shown in FIG. 5 and described below.

[0036] The computing device 102 can deploy one of the SNNs from the sweep on the neuromorphic processor 124. As an example, the computing device 102 can select the SNN that performs best in terms of accuracy or other metrics of speech recognition using the hyper-parameter settings selected for the audio-to-spike transform pipeline 122.

[0037] 2 illustrates four stages 142-148 of an exemplary audio-spike transform pipeline 122. The four stages may include, by way of example, a sampling stage 142, a frequency domain stage 144, a spectrogram stage 146, and a spiking representation stage 148. The processing of each stage may be performed by a separate component of the audio-spike transform pipeline 122, such as a separate engine, software module, or processor of the audio-spike transform pipeline 122. By way of example, the sampling stage 142 may be performed by a sampling engine, the frequency domain stage 144 may be performed by a frequency domain engine, the spectrogram stage 146 may be performed by a spectrogram engine, and the spiking representation stage 148 may be performed by a spiking representation engine. The processing of stages 142-148 may be performed in software and / or on a hardware accelerator, either sequentially or in parallel.

[0038] In the sampling stage 142, the audio-to-spike transform pipeline 122 begins by acquiring audio data 210, which may be in the form of raw audio or pre-processed audio. Examples of pre-processing may include reducing noise in the audio data, for example, using a filtering mechanism or by sound image localization. The audio data 210 may be acquired from a file or memory, or from an active microphone. In some examples, the computing device 102 may acquire the audio data 210 at a predefined sampling rate. By way of example, the raw audio may be 16KHz, 16-bit raw audio. The audio data 210 may be acquired at other sampling rates and / or bit-precision recordings.

[0039] During the sampling stage 142, the audio data 210 may be centered and / or scaled. As an example, the audio spike transform pipeline 122 may center the audio data 210 by subtracting an estimate of the bias and scaling to achieve a peak-to-peak amplitude threshold, such as a peak-to-peak amplitude of 0.5 or another suitable value. The audio spike transform pipeline 122 may center and scale the audio data 210 on a sample-by-sample basis. In some instances, the bias is often small enough to be negligible.

[0040] In the frequency domain stage 144, the audio spike transform pipeline 122 transforms the audio data into an audio signal in the frequency domain. The audio spike transform pipeline 122 may perform this transformation using a Fast Fourier Transform (FFT), a Sliding Discrete Fourier Transform (SDFT), or another suitable mechanism for obtaining a frequency domain representation of the audio data 210. As an example, the audio spike transform pipeline 122 may apply an SDFT to the audio data 210 to encode the audio data into the frequency domain and perform a Fourier decomposition of the incoming audio, mimicking the way the cochlea functions as a filter bank. The SDFT is an incremental method for computing the power spectrum of the audio data 210. In some cases, the incremental computation may be faster than a full FFT, for example, depending on the hop size of the FFT. Additionally, the SDFT may operate with constant time complexity per sample through parallelization and hardware acceleration.

[0041] The output of the frequency domain stage 144 may include a frequency domain representation of the audio data 210, which may include instantaneous representations of the spectral content of the audio signal, which represent the power of the frequencies that make up the audio signal. These instantaneous representations may be added in sequence to generate the spectrogram 220.

[0042] In the spectrogram stage 146, the audio spike transform pipeline 122 may map the frequency domain representation of the audio data 210 to a set of frequency bands, resulting in a spectrogram 230. As an example, the audio spike transform pipeline 122 may map the frequency domain representation of the audio data 210 to a set of mel-scale frequency bands, resulting in a log-mel spectrogram 230. The audio spike transform pipeline 122 may again logarithmically transform, center, and scale the frequency domain audio signal. As an example, to generate the log-mel spectrogram 230, the audio spike transform pipeline 122 may map the frequency domain output by the frequency domain stage 144 to a mel-scale representing how humans perceive pitch in speech, and apply another transform using overlapping windows with specific shapes (e.g., triangular, cosine, etc.) and widths to non-linearly weight the frequencies according to the level of human perception. After weighting the frequencies in this way, the audio spike transform pipeline 122 may take the logarithm of the output to obtain a log-mel spectrum. Mel-scale frequency bands, or mel-features, are frequency features within the range of human speech. By mapping a frequency domain (e.g., SDFT features) to mel-scale frequency bands, the audio-to-spike transform pipeline 122 can generate mel-features that represent the human speech portion of the audio data 210, for example by emphasizing frequencies that humans are good at distinguishing.

[0043] In some implementations, the spectrogram stage 146 may be optional. In some examples, after computing the frequency domain representation, the power spectrum of the frequency domain audio signal is transformed into a number of mel-scale frequency bands (e.g., 10, 20, 30, or another suitable number of mel-scale frequency bands), and then a logarithmic transformation is applied to the transformed audio signal. At this point, in the audio spike transform pipeline 122, the data is a logarithmic mel spectrum with a time resolution equal to the audio sampling rate. In some implementations, the audio spike transform pipeline 122 can continue at this resolution. When training an SNN using Back-Propagation-Through-Time (BPTT), the number of time steps in each processed spike raster can drastically affect the overall training time. To avoid such effects, the audio spike transform pipeline 122 can downsample the spectrum to time steps, such as 5, 10, 20 time steps, or another suitable number of millisecond (ms) time steps.

[0044] In some implementations, the audio spike transform pipeline 122 stacks the log-mel signal multiple times. As an example, after downsampling, 20 log-mel signals can be stacked four times, resulting in a total of 80 log-mel signals. Here, stacking refers to concatenating individual features with log-mel features, e.g., over time. As an example, these features can include features obtained from the log-mel representation, such as a cumulative sum of features over time. In another example, separate arrays can be concatenated into one array, e.g., repeating the same signal represented as an array multiple times.

[0045] In the spiking representation stage 148, the audio-spike conversion pipeline 122 converts the log-mel signals, such as stacked log-mel signals, into a spiking representation 240. The audio-spike conversion pipeline 122 can convert the stacked log-mel signals into spikes using a step-forward algorithm. Figures 4A-4B show an example process for generating spikes using the step-forward algorithm. The step-forward algorithm converts continuous-valued signals into spikes.

[0046] The step forward algorithm may include one or more of several types of delta modulation, sometimes also referred to as differential pulse code modulation. In one type, which may be referred to as discrete differentiation, successive analog signals are quantized and the difference between successive signals is determined. In another type, which may be referred to as an integral value approach, the difference is taken with respect to an integrator, which is essentially a value that tracks the signal over time. That is, instead of the difference between two successive signals, the difference between the instantaneous value and the integral value of the signal over time is used. Both types of delta modulation may work in the audio spike pipeline 122, but the second type, which uses an integral signal, may provide improved performance compared to the discrete differentiation approach.

[0047] In the differential differential approach, a step-by-step difference between consecutive signals can be calculated. When the step-by-step difference meets a threshold in either the positive or negative direction, a spike of the corresponding polarity is generated. By calculating the step-by-step difference, a set of spikes (positive spikes and / or negative spikes) can be generated. The spikes can represent changes in audio information (e.g., a log-Mel signal). The spikes can represent the quantized non-negative discrete derivative of the log-Mel signal.

[0048] In the integral approach, the integral of the signal can be calculated over time. For each new signal, the difference between the new instantaneous value of the signal and the integral is calculated. Similar to the differential difference approach, a set of spikes can be generated based on this difference.

[0049] In addition to these sets of difference-based spikes, the audio-spike transform pipeline 122 can generate another set of spikes based on the cumulative sum of the log-mel signals, e.g., cumulative sum spikes. The cumulative sum spikes can be calculated by calculating a quantized sum of each log-mel signal. The cumulative sum spikes can represent a spike encoding of the sustained value of the audio information (e.g., the log-mel signals). As shown in FIG. 2, the spikes generated for the audio 210 can include stepwise difference spikes 242 and cumulative sum spikes 244. The generated spikes can be processed by an SNN for speech recognition.

[0050] In some implementations, the hyperparameters used in various stages of the audio-to-spike transform pipeline 122 can be determined by the hyperparameter tuner 128. By way of example, the hyperparameters can include spike encoding method, threshold, hop length (ms), time step (ms), log-spectral features, spectral features, number of channels, and other suitable hyperparameters. The hyperparameter tuner can perform a hyperparameter sweep using multiple sets of hyperparameter settings and select the hyperparameter setting that results in the best SNN accuracy. By way of example, the hyperparameter tuner can perform a sweep on three different spike encoding methods, including the delta method, the rate method, and the step-forward method. Based on the results of the sweep, the hyperparameter tuner can determine that the best spike encoding method is the step-forward method.

[0051] 3 is a flow diagram of an example process 300 for converting audio data into spikes and processing the spikes using an SNN to recognize speech sounds. In some implementations, at least a portion of the process 300 can be performed on the computing device 102.

[0052] At step 302, a computing device acquires audio data. As an example, the computing device may acquire real-time audio data from a microphone. The microphone may be located internal or external to the computing device. In some examples, the computing device may acquire the audio data from a storage device or memory of the computing device.

[0053] In some examples, the computing device can capture audio data at a predetermined sampling rate. As an example, the captured audio data can be 16KHz, 16-bit raw audio. Other frequencies and bit-precision recordings can also be used.

[0054] In some implementations, the raw audio data can be centered and scaled. For example, a computing device can center the raw audio data by subtracting an estimate of the bias (which in practice is often negligibly small) and scale the raw audio to achieve a predetermined peak-to-peak amplitude, such as 0.5, 0.7, 1.0, or another suitable value.

[0055] At step 304, the computing device transforms the audio data (e.g., the centered and scaled audio data) into the frequency domain to obtain a frequency domain audio signal. To encode the audio data into the frequency domain, a Fourier transform can be applied to the audio data. As an example, a SDFT can be applied to incrementally calculate the Fourier transform of the audio data as follows:

[0056]

number

[0057] During the ceremony,

[0058]

number

[0059] is the sample f t-M+1 ,f t-M+2 ,…,f t is the frequency of the DFT in the kth bin at time step t, calculated from 0 ≤ k ≤ M-1, t is a time step in a sliding window of length T, and G = e j2π / M is a scaling factor.

[0060] This incremental computation can be much faster than the typical method for performing continuous speech recognition using the FFT, where the audio signal is sliced ​​into overlapping windows and the DFT is computed for each window. Furthermore, the SDFT can operate with constant time complexity per sample through parallelization and hardware acceleration.

[0061] In step 306, the computing device maps the frequency domain audio signal to a set of mel-scale frequency bands to obtain a logarithmic mel signal. Specifically, the frequency domain audio signal can be mapped to a set of mel-scale frequency bands to obtain a mel-scale frequency audio signal. The mel-scale frequency bands or mel features are frequency features in the range of human speech. The mel-scale frequency audio signal can then be logarithmically transformed to obtain a logarithmic mel signal.

[0062] In some implementations, the computing device may downsample the log-mel signal, which may preserve performance while still allowing the audio-to-spike transform pipeline 122 to run in real time. By way of example, the log-mel signal may be downsampled to a time step of 5 ms or other suitable time step.

[0063] The computing device may stack, e.g., concatenate, the log-mel signals a predetermined number of times. As an example, after computing the SDFT, the frequency domain audio signal is converted into several (e.g., 5, 10, 20, 30, etc.) mel-scale frequency bands. The log-mel signals may be stacked four times to result in four blocks of log-mel signals totaling 80 units, with each block containing 20 units of log-mel signals. For example, the four blocks may represent a log-mel spike, a cumulative sum of the log-mel spikes, and two channels for each of them. When binary data is transmitted, there is no way to represent negative numbers. Thus, negative and positive changes are split into two separate channels. Other suitable numbers of blocks may be used. In some implementations, each log-mel signal in the stack may be centered and scaled using static estimates of the mean and standard deviation obtained from a trail run of data-driven feature extraction.

[0064] At step 308, the computing device converts the log-mel signal to spikes. The conversion of the log-mel signal to spikes can be using a step-forward algorithm. This algorithm can encode continuous-valued signals into spikes on multiple channels by taking step-by-step differences between successive signals. When a step-by-step difference meets (e.g., exceeds) a threshold in either the positive or negative direction, a spike of the corresponding polarity is generated. Step-forward is useful for neuromorphic speech recognition systems because it can be implemented efficiently on a general-purpose processor for a reasonable number of channels.

[0065] 4A-4B show an example process of generating spikes on an audio signal. FIG. 4A shows the step-forward algorithm using an integral approach applied to two streaming signals: an artificial input 410 and a raw audio recording 420. For each streaming signal, there is a residual signal representing an integral generated by delta modulation based on integrals. As an example, for a raw input 422 of a raw audio recording, there is a residual signal 424 representing an integral. Using the residual signal, the two streaming signals 410 and 420 are converted into spikes. As an example, the artificial input 410 is converted into a positive spike 412 and a negative spike 414. The raw audio recording 420 is converted into a positive spike 426 and a negative spike 428. The step-forward algorithm converts the continuous-valued signal into a binary signal that can be interpreted as spikes by quantizing the signal, taking the difference between the raw input 422 and the residual signal 424, and applying a threshold.

[0066] As an example, the instantaneous value of the raw input 410 can be compared to the residual signal 424. The difference between these two signals is then used to generate the spikes. The residual signal 424 is then modified by the quantized result. As an example, if the instantaneous value of the raw signal 410 is greater than the residual signal 424, the residual signal 410 is increased. Similarly, if the instantaneous value of the raw signal 410 is less than the residual signal 424, the residual signal 410 is decreased.

[0067] Figure 4B shows the process of generating spikes using the step-forward algorithm. In this example, spikes are generated for the raw audio recording streaming signal 420 of Figure 4A. For a raw input 422 of the raw audio recording, there is a residual signal 424 that represents the integral of the signal over time.

[0068] A difference between the continuous raw input signal 422 and the residual signal 424 can be calculated. The difference between the log-mel signals of the two signals is determined. If the difference meets a threshold in either the positive or negative direction, a spike of the corresponding polarity is generated. In other words, if the absolute value of the difference meets a threshold, a positive or negative spike can be generated. As an example, a residual signal X1 430 and a real signal Y1 432 are shown. The residual signal X1 is compared with the real signal Y1, and if the difference is greater than the threshold, it is quantized. As an example, if the difference X1-Y1 meets (e.g., is smaller than) a negative threshold -θ 434, a negative spike 436 can be generated. Similarly, if the difference between the residual X2 and the real signal Y2 meets (e.g., is larger than) a positive threshold θ 444, a positive spike 446 can be generated.

[0069] By calculating the difference, a set of spikes (positive and / or negative spikes) can be generated. The spikes can represent changes in the audio information (e.g., log-mel signal). That is, changes or new information can be captured and processed only if there is a change or new information is available. In this way, the audio information to be processed is reduced, thus enabling speech recognition on power-constrained devices.

[0070] In addition to the set of difference-based spikes, the computing device may generate spikes based on a cumulative sum of the log-Mel signals, e.g., cumulative sum spikes. The cumulative sum spikes may be determined by calculating a quantized sum of each log-Mel signal. As an example, a cumulative sum may be applied to each log-Mel signal over time to obtain a cumulative sum spike. The cumulative sum spikes may represent a spike encoding of a sustained value of the audio information (e.g., the log-Mel signal). In some implementations, the cumulative sum spikes may be concatenated with the difference spikes. As an example, the spikes for the first and second blocks (e.g., units 0 to 39) may represent a difference spike, e.g., a discrete derivative of a quantized, continuous signal value. The spikes in the third and fourth blocks (e.g., units 40 to 79) may represent a cumulative sum spike, e.g., a sustained value of the log-Mel signal rather than just a change. Multiple sets of spikes, including step-by-step difference spikes (e.g., positive signal spikes and negative signal spikes) and cumulative sum spikes (e.g., positive cumulative spikes and negative cumulative spikes), can be input to an SNN for speech recognition.

[0071] In some implementations, the log-mel features and the cumulative sums can be provided together as inputs to the spiking representation stage 148. Each channel can then be converted separately to spikes.

[0072] At step 310, the computing device can process the spikes using an SNN for speech recognition. As an example, the computing device can run an SNN using the spikes as input. The SNN can process the spikes to generate speech recognition results including text, e.g., words or phonemes, corresponding to sounds in the audio data. As an example, the SNN can predict speech sounds in the form of phonemes based on the spikes received from the audio-to-spike transform pipeline 122. The phonemes can then be processed by another component, e.g., a lexicon and / or a language model, to generate words or sentences corresponding to the phonemes.

[0073] The computing device or another device can perform an action based on the speech recognition result. As an example, an AI assistant can perform an action based on a word or sentence (e.g., play a song, order an item, etc.). The speech recognition result can be displayed in a user interface of the user device. In some examples, the computing device can be a user device. In some other examples, the computing device can be a device different from the user device.

[0074] The order of steps in process 400 described above is exemplary only, and process 400 can be performed in a different order. In some implementations, process 400 can include additional steps, fewer steps, or some steps can be divided into multiple steps.

[0075] The techniques described in this document can be used to train an SNN using the described audio-to-spike transform pipeline 122. As an example, a feed-forward SNN architecture can include an input layer of 80 (or another suitable number) neurons taking as input 1 second of audio samples encoded into spikes by the audio-to-spike transform pipeline 122 over 200 time steps, three hidden layers of 256 leaky integrate-and-fire (LIF) neurons each, and an output layer of 35 LIP neurons or 20 LIF neurons. As an example, the input layer can include a number of layers equal to the number of stacked Mel signals / features.

[0076] In some implementations, SNNs can be trained using snnTorch, which enables efficient surrogate gradient learning for SNNs. The model can be trained using spike rate outputs, where output units corresponding to the target class can be given a high target spike rate (e.g., 50 spikes / sec) and non-target output units can be given a low target spike rate (e.g., 5 spikes / sec). The training loss can be calculated by snnTorch using the mean squared error between the actual spike rate and the target spike rate. The loss can be backpropagated using fast sigmoid surrogate gradients, and the parameters can be updated using an Adam optimizer. The model's output class can be determined by selecting the index of the output unit with the highest total number of spikes for the recording.

[0077] In some implementations, the system can train the SNN model using a hardware-aware training method. For example, the system can train the SNN model using a feed-forward five-layer architecture using SLAYER and a functional neuromorphic hardware simulator, such as the simulator of Intel's Loihi chip. SLAYER is a surrogate gradient training framework built on pyTorch that supports Current-Based Leaky Integrate-and-Fire (CUBA LIF) neuron models and provides a straightforward, hardware-aware training process to simplify the deployment of trained models. SLAYER trains fill-precision shadow weights that are quantized during the forward inference stage to the same precision supported by the target hardware.

[0078] The techniques described in this document can determine hyperparameters for the audio spike transform pipeline 122. In some implementations, the system can interact with a platform that allows for distributed model optimization while keeping a shared record of all parameters and performance metrics. The system can perform a hyperparameter sweep to identify the best combination of hyperparameters for the audio spike transform pipeline 122. Instead of performing real-time transformation and inference for the sweep, in offline training, the system can use a standard FFT in the audio spike transform pipeline 122, which is more efficient for batch processing than the SDFT that the pipeline typically uses, since it allows for low single sample latency for streamed audio. The sweep can generate an estimate of the relative importance of each hyperparameter. The relative importance is the degree to which each hyperparameter was useful in predicting the chosen metric. The importance can be calculated from feature importance values ​​of a random forest trained using the hyperparameters as inputs and the metric as the target output. Classification accuracy can be chosen as the metric to optimize for the sweep. In some implementations, the hyperparameters may include spike transform, threshold, hop length (ms), time step (ms), log-spectral features, spectral features, number of channels, and other suitable hyperparameters.

[0079] The audio-to-spike transform pipeline 122 is an efficient audio processing pipeline for real-time neuromorphic speech recognition systems. SNNs trained on data output by the audio-to-spike transform pipeline can achieve high accuracy in keyword classification. The pipeline can provide more suitable spiking benchmarks for speech recognition. The efficiency of the pipeline can enable a larger number of spiking speech recognition benchmarks in neuromorphic speech recognition.

[0080] Pipelines are more suitable for real-time operation and are suitable for use in real-time speech recognition systems. The techniques described in this document may lead to a wider adoption of neuromorphic accelerators in edge devices to provide voice control, which may have knock-on effects on the intelligence that can be successfully integrated into these devices. Neuromorphic accelerators can provide several application-level advantages, including continuous learning and multi-modal processing.

[0081] 5 is a flow diagram of an example process 500 for tuning hyperparameters of an audio spike transform pipeline. In some implementations, at least a portion of the process 500 can be performed by a computing device configured to train and assess the performance of a machine learning model, such as server 110 or server 112. For simplicity, the process 500 is described as being performed by a server.

[0082] At step 502, the server obtains a plurality of sets of hyperparameter settings. In some implementations, the hyperparameter settings include setting a set of hyperparameters with different values. As an example, one hyperparameter may be a time step. The time step settings may include 4 ms, 8 ms, 16 ms, 32 ms, 64 ms, and 128 ms. The set of hyperparameters may include a spike encoding method, a threshold, a hop length (ms), a time step (ms), a log-spectral feature, a spectral feature, a number of channels, and other suitable hyperparameters.

[0083] At step 504, the server performs a sweep of each hyper-parameter setting. Each sweep can include applying a particular hyper-parameter setting to the audio-to-spike transform pipeline (step 506), training a plurality of SNNs on the neuromorphic processor using the training data and spikes generated by the audio-to-spike transform pipeline based on the training data (step 508), testing each of the plurality of SNNs using the test data (step 508), and determining the accuracy of each SNN based on the testing (step 510). The training data and / or the test data can include labeled audio data with labels indicative of speech sounds, e.g., words or phonemes, corresponding to the audio data.

[0084] The server may perform steps 506-510 multiple times to identify hyperparameters and a trained machine learning model to deploy on the audio spike transform pipeline 122. By way of example, the server may perform steps 506-510 until it finds a trained machine learning model that achieves at least a minimum level of accuracy.

[0085] At step 512, the server determines the relative importance of the hyperparameters. The server may determine the relative importance of each hyperparameter based on the results of each sweep. By way of example, if there is a significant change in the accuracy of the SNN trained during a sweep when the setting of a particular hyperparameter is adjusted, then that particular hyperparameter is likely to have a large impact on the accuracy of the SNN and is therefore important. The server may evaluate various factors in assessing the importance of each hyperparameter, such as the change in accuracy of the SNN with changes to the hyperparameter, whether the hyperparameter contributed to the most accurate SNN or at least a threshold number of the most accurate SNNs, and / or other suitable factors.

[0086] In some implementations, the server can use the importance of each hyperparameter in step 506 to determine which hyperparameters to test using steps 506-510. As an example, if a hyperparameter is found to be insignificant for extensive testing (e.g., at least a threshold number of tests), the server can remove the process corresponding to that hyperparameter from the pipeline. In another example, the server can adjust one or more of the hyperparameters based on their importance.

[0087] At step 514, the server finalizes the hyperparameters of the audio-to-spike transform pipeline 122 based on the accuracy of the SNN and / or the importance of the hyperparameters. The computing device may stop using one or more hyperparameters based on the relative importance of the hyperparameters. The server may also adjust the settings of one or more hyperparameters based on the accuracy and / or importance. Processing the audio data or intermediate forms of the audio data based on each hyperparameter has an associated computational cost (e.g., CPU cycles) and latency. By removing those with less importance, the conversion of the audio data to spikes can be performed faster and more efficiently with little or no adverse effect on the accuracy of the SNN's speech recognition results. It should be noted that some hyperparameters may have more importance for some languages ​​and / or dialects. Thus, the ability to use a large number of hyperparameters and filter some based on importance allows for highly accurate speech recognition with a highly efficient speech-to-spike transform.

[0088] In some implementations, the computing device selects which hyperparameters to use or not to use based on the relative importance and / or computational cost of each hyperparameter. For example, if a hyperparameter is of medium importance but has the highest computational cost, the hyperparameter tuner 128 may turn off that hyperparameter to increase the efficiency of the conversion to spikes.

[0089] Certain novel aspects of the subject matter herein are set forth in the appended claims.

[0090] The subject embodiments and functional operations described herein can be implemented in digital electronic circuitry, or in tangibly embodied computer software or firmware, or in computer hardware including the structures disclosed herein and their structural equivalents, or in any combination of one or more of them. The subject embodiments described herein can be implemented as one or more computer programs, i.e., as one or more modules of computer program instructions encoded on a tangible, non-transitory program carrier for execution by or control of the operation of a data processing device. Alternatively, or in addition, the program instructions can be encoded in an artificially generated propagated signal, e.g., a machine-generated electrical, optical, or electromagnetic signal, that is generated to encode information to be transmitted to a suitable receiver device for execution by the data processing device. The computer storage medium can be a machine-readable storage device, a machine-readable storage substrate, a random access memory device, or a serial access memory device, or a combination of one or more of them. It is noted that the computer storage medium is not a propagating signal.

[0091] The term "data processing apparatus" encompasses all types of apparatus, devices, and machines that process data, including, by way of example, a programmable processor, a computer, or multiple processors or computers. An apparatus may include special purpose logic circuitry, such as, for example, a field programmable gate array (FPGA) or an application specific integrated circuit (ASIC). In addition to hardware, an apparatus may also include code that creates an execution environment for a subject computer program, such as code that constitutes, for example, processor firmware, a protocol stack, a database management system, an operating system, or a combination of one or more of these.

[0092] A computer program (which may also be referred to or described as a program, software, software application, module, software module, script, or code) may be written in any form of programming language, including compiled or interpreted, or declarative or procedural, and may be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment. A computer program may, but need not, correspond to a file in a file system. A program may be stored as part of a file that holds other programs or data, e.g., one or more scripts stored in a markup language document, in a single file dedicated to the program in question, or in multiple associated files, e.g., multiple files that store one or more modules, subprograms, or pieces of code. A computer program may be deployed to be executed on one computer, or on multiple computers located at one site or distributed across multiple sites and interconnected by a communications network.

[0093] As used herein, "engine" or "software engine" refers to a software-implemented input / output system that provides an output distinct from an input. An engine can be a block of encoded functionality, such as a library, a platform, a software development kit ("SDK"), or an object. Each engine can be implemented on any suitable type of computing device, such as, for example, a server, a mobile phone, a tablet computer, a notebook computer, a music player, an e-reader, a laptop or desktop computer, a personal digital assistant (PDA), a smartphone, or other stationary or portable device that includes one or more processors and a computer-readable medium. Furthermore, two or more of the engines may be implemented on the same computing device or on separate computing devices.

[0094] The processes and logic flows described herein may be performed by one or more programmable computers executing one or more computer programs that perform functions by operating on input data and generating output. The processes and logic flows may also be performed by, and an apparatus may be implemented as, special purpose logic circuitry, such as, for example, an FPGA (field programmable gate array), an ASIC (application specific integrated circuit), a neuromorphic research chip such as Intel's Loihi chip, or a neural network processor such as BrainChip's Akida™ chip.

[0095] A computer suitable for executing a computer program can be based on and includes, for example, a general-purpose or dedicated microprocessor or both, or any other type of central processing unit. In general, the central processing unit receives instructions and data from a read-only memory or a random access memory or both. The essential components of a computer are a central processing unit for implementing or executing instructions, and one or more memory devices for storing instructions and data. In general, a computer also includes one or more mass storage devices for storing data, such as, for example, magnetic disks, magneto-optical disks, or optical disks, and / or is operatively coupled to receive or transfer data therefrom. However, a computer may not have such devices. Furthermore, a computer can be incorporated into another device, such as, for example, a mobile phone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a Global Positioning System (GPS) receiver, or a portable storage device, such as, for example, a universal serial bus (USB) flash drive, to name just a few.

[0096] Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and memory devices, including, by way of example, semiconductor memory devices such as EPROM, EEPROM, and flash memory devices, magnetic disks such as internal hard disks or removable disks, magneto-optical disks, and CD ROM and DVD ROM disks. The processor and memory can be supplemented by, or incorporated in, special purpose logic circuitry.

[0097] To provide for interaction with a user, embodiments of the subject matter described herein can be implemented on a computer having a display device, such as, for example, a cathode ray tube (CRT) monitor, a liquid crystal display (LCD) monitor, or an OLED display, for displaying information to a user, and an input device, such as, for example, a keyboard, a mouse, or a presence-sensitive display or other surface, for providing input to the computer. Other types of devices can also be used to provide for interaction with a user. By way of example, feedback provided to the user can be any form of sensory feedback, such as, for example, visual feedback, auditory feedback, or haptic feedback, and input from the user can be received in any form, including acoustic, speech, or tactile input. Additionally, a computer can interact with a user by sending and receiving resources to and from a device used by the user, such as, for example, by sending a web page to a web browser in response to a request received from the web browser on the user's client device.

[0098] Embodiments of the subject matter described herein may be implemented in a computing system that includes a back-end component, e.g., as a data server, or a computing system that includes a middleware component, e.g., an application server, or a computing system that includes a front-end component, e.g., a client computer having a graphical user interface or a web browser through which a user can interact with an implementation of the subject matter described herein, or any combination of one or more such back-end, middleware, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication, e.g., a communications network. Examples of communications networks include local area networks ("LANs") and wide area networks ("WANs"), e.g., the Internet.

[0099] A computing system may include clients and servers. Clients and servers are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other.

[0100] Although the present specification contains many specific implementation details, these should not be construed as limitations on the scope of any invention or the scope of the claims, but rather as descriptions of features that may be specific to certain embodiments of a particular invention. Certain features described in the present specification in the context of separate embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented in multiple embodiments separately or in any suitable subcombination. Furthermore, although features may be described above as operating in a particular combination, and may even be initially claimed as such, in some cases one or more features of a claimed combination may be deleted from the combination, and the claimed combination may be directed to a subcombination or a variation of the subcombination.

[0101] Similarly, although operations are shown in a particular order in the figures, this should not be understood as requiring that the operations be performed in the particular order or sequence shown, or that all of the operations shown be performed, to achieve desirable results. In certain situations, multitasking and parallel processing may be advantageous. Furthermore, the separation of various system modules and components in the above-described embodiments should not be understood as requiring such separation in all embodiments, and it will be understood that the program components and systems described may generally be integrated into a single software product or packaged into multiple software products.

[0102] Specific embodiments of the subject matter have been described. Other embodiments are within the scope of the following claims. By way of example, the actions recited in the claims may be performed in a different order and still achieve desirable results. By way of example, the processes depicted in the accompanying figures do not necessarily require the particular order or sequence depicted to achieve desirable results. In certain implementations, multitasking and parallel processing may be advantageous.

Claims

1. 1. A method performed by one or more data processing apparatus using one or more components of an audio spike transformation pipeline, comprising: obtaining audio data; generating a frequency domain audio signal representative of said audio data by transforming said audio data into the frequency domain; mapping said frequency domain audio signal into a set of Mel frequency bands to obtain a Mel scale frequency audio signal; performing a logarithmic transformation on said Mel-scale frequency audio signal to obtain a logarithmic Mel-signal; generating a spike input comprising a series of spikes to be input to a spiking neural network (SNN) model by converting the log-mel signal into the series of spikes, wherein converting the log-mel signal into spikes comprises calculating a quantized sum of each log-mel signal to determine a running sum spike; providing said spike input as an input to said SNN model; The method comprising:

2. executing the SNN model using the spike input, the spiking neural network processing the series of spikes to generate a speech recognition result including text corresponding to the audio data; outputting the speech recognition result; The method of claim 1 further comprising:

3. The method of claim 2 , further comprising the step of performing an action based on the speech recognition result.

4. The step of acquiring audio data includes: retrieving the audio data from a memory; or obtaining said audio data from an active microphone; The method of claim 1 , comprising:

5. Transforming the audio data into the frequency domain comprises: centering and scaling the audio data to obtain centered and scaled audio data; transforming the centered and scaled audio data into the frequency domain using a sliding window of a predetermined length; The method of claim 1 , comprising:

6. The method of claim 1 , wherein transforming the audio data into the frequency domain comprises transforming the audio data using a sliding discrete Fourier transform (SDFT) algorithm.

7. The step of converting the logarithmic mel-signal into a spike input comprises: determining a step-by-step difference between the log-mel signals of two adjacent time steps; determining whether the stepwise difference meets a threshold in either a positive or negative direction; generating a spike of a corresponding polarity; The method of claim 1 , comprising:

8. Transforming the logarithmic mel-signal into spikes comprises: stacking the log-mel signals multiple times to obtain stacked features in multiple blocks; converting the stacked features in the plurality of blocks into spikes; The method of claim 1 , comprising:

9. The method of claim 1 , wherein the one or more components of the audio spike transform pipeline include hyper-parameters, the hyper-parameters being tuned based on the accuracy of the SNN model and / or the importance of the hyper-parameters.

10. For each hyperparameter setting in the set of hyperparameter settings, applying the hyper-parameter settings to the one or more components of the audio spike transform pipeline; training a plurality of test SNN models using training audio data that is converted to spike inputs using the one or more components of the audio-to-spike transform pipeline; and testing each test SNN model using test data, the testing step including generating result data by determining accuracy of speech recognition results generated by each test SNN model; for each set of hyperparameters, updating the hyperparameters based on the results data; The method of claim 9 further comprising:

11. 11. The method of claim 10, wherein the method further comprises, for each set of hyperparameters, determining an importance of each hyperparameter based on the results data, and wherein updating the hyperparameters comprises updating the at least one hyperparameter based on the importance of the at least one hyperparameter.

12. 12. The method of claim 11, further comprising avoiding use of a given hyper-parameter in generating the spike inputs based on the importance of the given hyper-parameter.

13. one or more processors, including a neuromorphic processor; one or more storage devices storing instructions that, when executed by the one or more processors, cause the one or more processors to perform operations; a system including: Obtaining audio data; generating a frequency domain audio signal representative of the audio data by transforming the audio data into the frequency domain; mapping said frequency domain audio signal into a set of Mel frequency bands to obtain a Mel scale frequency audio signal; performing a logarithmic transformation on said Mel-scale frequency audio signal to obtain a logarithmic Mel-signal; generating a spike input comprising a series of spikes to be input to a spiking neural network (SNN) model by converting the log-mel signal into the series of spikes, where converting the log-mel signal into spikes comprises calculating a quantized sum of each log-mel signal to determine a running sum spike; providing said spike input as an input to said SNN model; Including, the system.

14. The operation includes: executing the SNN model using the spike input, the spiking neural network processing the series of spikes to generate a speech recognition result including text corresponding to the audio data; outputting the speech recognition result; The system of claim 13 , comprising:

15. The system of claim 14 , wherein the action includes performing an action based on the speech recognition result.

16. The obtaining of the audio data includes: retrieving the audio data from a memory; or acquiring said audio data from an active microphone; The system of claim 13 , comprising:

17. Transforming the audio data into the frequency domain includes: centering and scaling the audio data to obtain centered and scaled audio data; Transforming the centered and scaled audio data into the frequency domain using a sliding window of a predetermined length. The system of claim 13 , comprising:

18. 14. The system of claim 13, wherein transforming the audio data into the frequency domain includes transforming the audio data using a sliding discrete Fourier transform (SDFT) algorithm.

19. One or more computer storage media storing instructions that, when executed by one or more processors, including a neuromorphic processor, cause the one or more processors to perform operations, the operations including: Obtaining audio data; generating a frequency domain audio signal representative of the audio data by transforming the audio data into the frequency domain; mapping said frequency domain audio signal into a set of Mel frequency bands to obtain a Mel scale frequency audio signal; performing a logarithmic transformation on said Mel-scale frequency audio signal to obtain a logarithmic Mel-signal; generating a spike input comprising a series of spikes to be input to a spiking neural network (SNN) model by converting the log-mel signal into the series of spikes, where converting the log-mel signal into spikes comprises calculating a quantized sum of each log-mel signal to determine a running sum spike; providing said spike input as an input to said SNN model; [0036] In one embodiment, the present invention relates to a computer-readable storage medium (100).

Citation Information

Patent Citations

  • Multiple sound discriminating apparatus

    JP2009230337A

  • Frequency analysis device and program

    JP2016090670A

  • Asynchronous pulse modulation for threshold-based signal coding

    JP2017526224A

  • Method and device for obtaining an event designation based on audio data

    JP2020524300A

  • Apparatus, system and method for varying synaptic weights with phase difference in spiking neural networks

    JP2021515283A