Processing Streaming Data

Abstract

Techniques for processing streaming data are disclosed. In some embodiments, the techniques include obtaining input data representing a frame of streaming data. The techniques may include identifying a query transformation, a key transformation, and a value transformation based on the input data. The techniques may include updating a query buffer, a key buffer, and a value buffer such that the buffers are configured to store a previous frame of the streaming data and parameters associated with the frame of the streaming data. The techniques may include reading one or more query frames from the query buffer. The techniques may include determining a dot product of the query frame and a frame in the key buffer to determine a weight set. The techniques may include determining a weighted sum between the weight set and a frame in the value buffer and generating a streaming attention vector using the weighted sum.

Description

[Technical Field]

[0001] [CROSS-REFERENCE TO RELATED APPLICATIONS] This application claims priority to U.S. Provisional Application No. 63 / 511,799, filed July 3, 2023, U.S. Provisional Application No. 63 / 416,429, filed October 14, 2022, and U.S. Provisional Application No. 63 / 401,046, filed August 25, 2022, each of which is incorporated herein by reference in its entirety.

[0002] [Technical field] The present disclosure relates to systems, methods, and media for processing streaming data. [Background technology]

[0003] Attention, for example, implemented as an attention vector, is a key component of machine learning algorithms and neural network architectures that have led to advances in natural language processing, computer vision, speech and audio processing, etc. For example, attention is implemented in transformer neural networks used in natural language processing, such as translating text from one language to another, outputting persuasive blocks of text in response to a query, etc. However, applying these algorithmic and neural network advances to streaming data, such as streaming audio data, is challenging due to the limited amount of data available in streaming contexts.

[0004] <Notes and Terms> Throughout this disclosure, including the claims, the terms "speaker," "loudspeaker," and "audio reproduction transducer" are used interchangeably to refer to any sound-emitting transducer (or set of transducers). A typical headphone set includes two speakers. Speakers may be implemented to include multiple transducers (e.g., woofers and tweeters), which are driven by a single common speaker feed or multiple speaker feeds. In some examples, the speaker feeds may undergo different processing in different circuit branches coupled to different transducers.

[0005] Throughout this disclosure, including the claims, the phrase performing an operation "on" a signal or data (e.g., filtering, scaling, transforming, or applying a gain to the signal or data) is used broadly to indicate performing an operation directly on the signal or data, or on a processed version of the signal or data (e.g., a version of the signal that has undergone preliminary filtering or post-processing before the operation is performed).

[0006] Throughout this disclosure, including the claims, the term "system" is used broadly to denote an apparatus, a system, or a subsystem. For example, a subsystem that implements a decoder may be referred to as a decoder system, and a system that includes such a subsystem (e.g., a system that generates X output signals in response to multiple inputs, of which the subsystem generates M inputs and the other XM inputs are received from external sources) may also be referred to as a decoder system.

[0007] Throughout this disclosure, including the claims, the term "processor" is used broadly to denote a system or device that is programmable or otherwise configurable (by software or firmware) to perform operations on data (e.g., audio or video or other image data). Examples of processors include field programmable gate arrays (or other configurable integrated circuits or chipsets), digital signal processors programmed and / or otherwise configured to perform pipeline processing on audio or other sound data, programmable general-purpose processors or computers, and programmable microprocessor chips or chipsets. Summary of the Invention

[0008] Techniques for processing streaming data are disclosed. In some embodiments, a method for processing streaming data includes obtaining input data representing a frame of streaming data. The method further includes identifying a query transformation, a key transformation, and a value transformation based on the input data for the frame of streaming data. The method includes updating a query buffer, a key buffer, and a value buffer based on the identified query, key, and value transformations, whereby the query buffer, the key buffer, and the value buffer are configured to store parameters associated with a previous frame of streaming data and the frame of streaming data, respectively. The method further includes reading one or more query frames from the updated query buffer for use in processing the input data. The method further includes determining a dot product of a frame in the key buffer and one or more query frames to determine a weight set. The method further includes determining a weighted sum between the weight set and a frame in the value buffer. The method includes generating a streaming attention vector using the weighted sum, the streaming attention vector usable by a network to generate a prediction associated with the streaming data.

[0009] In some examples, the network is a transformer network.

[0010] In some examples, the streaming data is streaming audio data.

[0011] In some examples, the prediction associated with the streaming data includes a prediction of a vocal emotion associated with the streaming data.

[0012] In some examples, the predictions associated with the streaming data include identifying one or more features useful for providing to one or more downstream machine learning models, hi some examples, the one or more features include identifying one or more speakers associated with the streaming data.

[0013] In some examples, the predictions associated with the streaming data include classifications of one or more words or phonemes in the streaming data.

[0014] In some examples, at least one of the query buffer, the key buffer, or the value buffer is a circular buffer.

[0015] In some examples, updating the query buffer comprises: adding a current query frame based on the query transformation to the query buffer; discarding the oldest query frame in the query buffer; In some examples, the one or more retrieved query frames correspond to the oldest discarded query frame.

[0016] In some examples, the network includes multiple layers, and updating the query buffer includes, for a first layer of the multiple layers: adding a current query frame based on the query transformation to the query buffer; replacing a plurality of query frames in the query buffer with a plurality of look-ahead query frames corresponding to future times; discarding the oldest query frame in the query buffer; In some examples, the one or more retrieved query frames used to process the input block by the first layer of the plurality of layers include the current query frame and the plurality of look-ahead query frames. In some examples, the plurality of look-ahead query frames include two look-ahead query frames. In some examples, the one or more retrieved query frames used to process the input block by each of the plurality of layers other than the first layer are passed to a given layer by a previous layer.

[0017] In some examples, updating the key buffer and updating the value buffer comprises: adding a current key frame to the key buffer based on the key transformation and discarding the oldest key frame from the key buffer; adding a current value frame to the value buffer based on the value transformation; discarding the oldest value frame from the value buffer; Includes.

[0018] In some examples, updating the key buffer and updating the value buffer comprises: adding a current key frame to the key buffer; replacing a plurality of key frames in the key buffer with a plurality of look-ahead key frames based on the key transformation; discarding the oldest key frame from the key buffer; adding a current value frame to the value buffer; replacing a plurality of value frames in the value buffer with a plurality of look-ahead value frames based on the value transformation; discarding the oldest value frame from the value buffer; Includes.

[0019] In some examples, the network comprises: performing initial training using a first version of the network that does not utilize the query buffer, the key buffer, and the value buffer to store a subset of query frames, key frames, and value frames; performing subsequent training that modifies weights associated with the network; In some examples, the subsequent training is performed using a second version of the network including the query buffer, the key buffer, and the value buffer, and performing the subsequent training includes performing backpropagation using derivatives derived from the second version of the network. In some examples, the subsequent training is performed using the first version of the network, and performing the subsequent training includes: providing the first version of the network with a given training data block of data, a series of time-shifted segments, to generate a corresponding series of predicted outputs; aggregating the set of predicted outputs; determining a loss based on the aggregated set of predicted outputs; and updating weights associated with the first version of the network based on the losses; and Includes.

[0020] Some or all of the operations, functions, and / or methods described herein may be performed by one or more devices according to instructions (e.g., software) stored on one or more non-transitory media. Such non-transitory media may include memory devices as described herein, including, but not limited to, random access memory (RAM), read-only memory (ROM), etc. Accordingly, various novel aspects of the subject matter described in this disclosure may be implemented via one or more non-transitory media having software stored thereon.

[0021] At least some aspects of the present disclosure may be implemented via an apparatus. For example, one or more devices may be capable of at least partially performing the methods disclosed herein. In some implementations, the device is or includes an audio processing system that includes an interface system and a control system. The control system may include at least one of a general-purpose single or multi-chip processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic element, discrete gate or transistor logic, discrete hardware components, or a combination thereof.

[0022] The details of one or more implementations of the subject matter described herein are set forth in the accompanying drawings and the description below. Other features, aspects, and advantages will become apparent from the description, drawings, and claims. The relative dimensions of the following drawings may not be drawn to scale. [Brief explanation of the drawings]

[0023] [Figure 1] FIG. 1 illustrates an example of an audio environment according to some embodiments.

[0024] [Figure 2] 1 is a flowchart illustrating an example process for training and using a machine learning model according to some embodiments.

[0025] [Figure 3A] 1 is a flowchart illustrating an example of a process for generating a prediction associated with an input signal according to some embodiments.

[0026] [Figure 3B] FIG. 1 illustrates the use of extracted features to generate predictions according to some embodiments.

[0027] [Figure 4] FIG. 1 is a block diagram of a portion of a transformer network that utilizes streaming attention according to some embodiments.

[0028] [Figure 5] 1 illustrates a streaming attention layer of a transformer network according to some embodiments.

[0029] [Figure 6] 1 illustrates components of a multi-head attention layer according to some embodiments.

[0030] [Figure 7] 1 illustrates the operation of a first-in, first-out (FIFO) delay manager according to some embodiments.

[0031] [Figure 8] 1 illustrates the operation of a history delay manager according to some embodiments.

[0032] [Figure 9] 1 illustrates components of a low-latency multi-head attention layer according to some embodiments.

[0033] [Figure 10A] 1 illustrates the operation of a FIFO delay manager that may be used in conjunction with a low-latency multi-head attention layer according to some embodiments. [Figure 10B]1 illustrates the operation of a FIFO delay manager that may be used in conjunction with a low-latency multi-head attention layer according to some embodiments.

[0034] [Figure 11] 1 illustrates the operation of a history delay manager that may be used in conjunction with a low-latency multi-head attention layer according to some embodiments.

[0035] [Figure 12] 1 is a flowchart illustrating an example process for generating streaming attention vectors according to some embodiments.

[0036] [Figure 13] 1 is a flowchart of an example process for pre-training a network according to some embodiments.

[0037] [Figure 14A] 1 is a flowchart of an example process for fine-tuning a pre-trained network with a custom kernel according to some embodiments.

[0038] [Figure 14B] 1 is a flowchart of an exemplary process for fine-tuning a pre-trained network using folding and unfolding techniques according to some embodiments.

[0039] [Figure 15] 1 illustrates an example implementation of a folding and unfolding technique for fine-tuning training in accordance with some embodiments.

[0040] [Figure 16] FIG. 1 is a block diagram illustrating example components of a device capable of implementing various aspects of the present disclosure.

[0041] Like reference numbers and designations in the various drawings indicate like elements. DETAILED DESCRIPTION OF THE INVENTION

[0042] Neural networks and other machine learning models have been used to generate predictions related to data. For example, with respect to audio data, trained machine learning models can be used to classify audio signals as speech or noise, perform speech recognition, and perform emotion detection on speech within the audio data. However, such networks have difficulty generating predictions for streaming data, where the data arrives continuously in real time or near real time. For example, Transformer networks have been used to perform speech recognition and text translation, among other functions, by utilizing a multi-head attention block that generates an attention vector based on all the data. As a more specific example, a conventional Transformer network can be used to generate an attention vector for a portion of an input string of speech or text that indicates the context of that portion relative to previous and future portions of the speech or text. However, this is not possible for streaming data, where future data (i.e., the rest of a sentence, the speech following a given word or phrase) is not readily accessible.

[0043] This specification discloses techniques (systems, methods, media, etc.) for implementing a network that utilizes streaming attention. In particular, the techniques described herein can be used to generate attention vectors based on a window of streaming data, rather than based on all data as is traditionally done in conventional Transformer networks. The attention vectors can be generated using a multi-head streaming attention block. Similar to conventional Transformer networks, the multi-head streaming attention block can be configured to generate attention vectors based on a query, a key, and a value transformation. The query transformation can generally indicate a set of vectors for which attention should be calculated, while the value transformation can generally indicate a set of vectors for which attention should be calculated. Thus, by performing a dot-product multiplication of the query by the value, a set of unnormalized weights can be generated that indicate how much attention each query is given to the key. The unnormalized weight set is then normalized, for example, using a softmax function, and the normalized weights are multiplied by the value to obtain an output. Unlike conventional multi-head streaming attention blocks of conventional Transformer networks, the multi-head streaming attention block can be configured to maintain and update query, key, and value buffers every frame, such that new query, key, and value transformations are updated in the buffers and old values ​​are discarded. As disclosed herein, unlike conventional Transformer networks that utilize all query data to generate attention vectors, the multi-head streaming attention block can use a window of query data to generate attention vectors.

[0044] In some embodiments, attention vectors can be generated using a frame inference method. In the frame inference method, query, key, and value buffers are updated based on the current frame of streaming data, and a single query frame is used to generate the attention vector. An example of a multi-head streaming attention block for implementing the frame inference method is described below in conjunction with FIG. 6. Additionally or alternatively, in some embodiments, attention vectors can be generated using a block inference method. In the block inference method, query, key, and value buffers are updated based on the current frame of streaming data and one or more look-ahead frames. The attention vector can be generated based on multiple query frames (e.g., the current query frame and one or more look-ahead frames). In conjunction with FIG. 9, an example of a multi-head streaming attention block (also referred to herein as a low-latency multi-head streaming attention block) for implementing the block inference method is described below. Note that the frame inference method can reduce computational complexity compared to the block inference method, while the block inference method can have low latency compared to the frame inference method. Therefore, a method can be selected based on the computational capabilities of the computing device used to generate inferences, the type of streaming data used, etc.

[0045] It should be noted that while the techniques disclosed herein are generally described with respect to streaming audio data, other types of streaming data are contemplated, and in some embodiments, other types of streaming data, such as streaming text data, may be used.

[0046] A network trained to make predictions related to streaming data can make any suitable predictions. For example, if the streaming data is streaming audio data, the predictions may include real-time or near-real-time predictions of speech recognition, emotion detection, speaker detection or identification, speech expression learning, etc. In some embodiments, generating predictions in real-time or near-real-time allows models to be updated without utilizing stored user data to generate the predictions or storing the data used for longer than necessary to generate the predictions. This can help protect user privacy, for example, by not storing audio or video conferencing data.

[0047] FIG. 1 illustrates an exemplary audio environment 100. FIG. 1 includes an audio device 104 that may be configured to capture an audio signal 101 via multiple microphones (e.g., microphones 106a, 106b, and 106c). An example of such an audio device is equipment for audio or video conferencing that may be used, for example, in a conference room. The audio device 104 may include one or more processors 105. The processor 105 may be configured to analyze the audio signal and, optionally, any other signal (e.g., a video signal, data from a motion sensor, etc.). The processor 105 may be configured to generate audio analytics, for example, to perform voice recognition, emotion recognition, speaker detection, etc. The audio analytics may be generated in real time or near real time based on streaming audio data. Techniques for performing or generating such audio analytics are disclosed herein that may be executed by the processor 105 or one or more processors communicatively coupled to the processor 105 (e.g., by a processor of a server device or other coupled computing device).

[0048] As mentioned above, disclosed herein are techniques for implementing streaming attention. This technique can be used with various types of neural network architectures, such as transformer networks, conformer networks, etc. Such neural networks can be trained, for example, using a training set. Once trained, a network utilizing streaming attention, as implemented via a multi-head streaming attention layer (described in more detail below), can be used in an inference stage to generate predictions for input streaming data. For example, the input streaming data can be streaming audio data (e.g., streaming speech signals, etc.). Predictions related to streaming audio data can include real-time or near-real-time speech recognition, speech emotion recognition, speaker detection or identification, speech expression learning, etc. In some embodiments, the streaming data can be streaming text data, video data, financial data, accelerometer data, power usage data, or any other type of time-series data.

[0049] It should be understood that training of a neural network configured to implement streaming attention may be performed by a device different from the device that utilizes the trained neural network for inference. For example, training may be performed by a first server device, and the trained neural network may be utilized for inference by an end-user device (e.g., an audio or video conferencing device, a laptop, a desktop computer, etc.), a second server device, etc. More detailed techniques for implementing streaming attention are shown in Figures 4, 5, 6, 7, 8, 9, 10A, 10B, 11, and 12 and described below. Techniques for training a network that utilizes streaming attention are shown in Figures 13, 14A, 14B, and 15 and described below.

[0050] FIG. 2 is a flowchart illustrating an example process for training and utilizing a trained machine learning model according to some embodiments. The machine learning model may implement streaming attention and be configured to generate predictions during the inference stage for streaming data. Note that in some embodiments, the blocks of process 200 may be implemented by multiple devices. For example, training the machine learning model may be implemented by a first device or set of devices, and using the machine learning model for inference may be implemented by a second device or set of devices. As another example, in some embodiments, training the model may be performed by one or more server devices, and inference may be performed by one or more client or end-computing devices, such as conferencing devices, laptop computers, desktop computers, etc. In some embodiments, the blocks of process 200 may be performed in a different order than that shown in FIG. 2. In some embodiments, two or more of the blocks of process 200 may be performed substantially in parallel. In some implementations, one or more blocks of method 200 may be omitted.

[0051] At 202, process 200 may obtain a training set. The training set may include training samples, each of which may include an input to a machine learning model and a corresponding ground truth prediction. For example, if a machine learning model is trained to perform speech recognition of an input speech signal, the training set may include training samples that each include a block of the input speech signal that includes speech and corresponding text indicating words contained in the input speech signal.

[0052] At 204, process 200 can train a machine learning model using the training set. The machine learning model may be a transformer network, a conformer network, etc. The machine learning model may have an architecture including a multi-head streaming attention layer or block, as described below in connection with FIGS. 5 and 6. Note that FIGS. 13, 14A, 14B, and 15 describe techniques for pre-training a model and then fine-tuning the model to generate a trained model. Furthermore, the trained machine learning model may be characterized by a weight set (e.g., weights associated with different nodes and / or layers of the network), where the weight set is the weight value at the end of the training phase.

[0053] At 206, process 200 may perform an inference stage by utilizing the trained machine learning model to make one or more predictions. The one or more predictions may be predictions associated with a block of streaming data, such as a block of streaming audio data. For example, the one or more predictions may include speech recognition associated with the block of streaming audio data, vocal emotion recognition associated with the block of streaming audio data, speaker detection associated with the block of streaming audio data, etc. Note that block 206 may be performed by a device different from the device that performed block 204. For example, block 206 may be performed by an end client device, such as a mobile user device, a desktop computer, or a conferencing device, while block 204 may be performed by a server device. As another example, block 206 may be performed by a cloud or server device different from the server device that performed the model training in block 204. For example, the cloud or server device may receive a block of streaming data from a client device (e.g., a video or audio conferencing device) and process the streaming data in the cloud to generate one or more predictions. In some implementations, the one or more predictions may be sent back to the client device.

[0054] Figures 3A and 3B illustrate inference steps applied to streaming data according to some embodiments. It should be noted that although Figures 2A and 2B generally illustrate streaming audio data, the techniques shown and described below may be applied to other types of streaming data.

[0055] Referring to FIG. 3A, components of an inference block 300 are shown according to some embodiments. As shown, an input audio signal may be provided to a feature extraction block 302. The input audio signal may be acquired from one or more microphones. The input audio signal may be streaming audio data, such that the audio arrives continuously in blocks, with each block containing a fixed number of samples. For example, the inference block 300 may operate at a sampling rate of 16,000 samples per second and receive 10 milliseconds (e.g., 160 samples) of audio at a time. Note that some systems may have a requirement that one block of audio must be processed within a block time. For example, if audio is received in 10-millisecond frames, each frame must be processed within 10 milliseconds to output predictions at the same rate as the input frames. The feature extraction block may be configured to extract a feature vector associated with the input acoustic audio signal.

[0056] The extracted feature vectors can be provided to an inference model 304. The inference model 304 can include a trained machine learning model that incorporates a streaming attention layer or block, as described below. The inference model 304 can generate, as output, one or more predictions associated with the input audio signal.

[0057] FIG. 3B shows a diagram of extracting feature vectors and providing the feature vectors to an inference model, according to some embodiments. As shown, at 352, an input audio signal is provided to a feature extraction block. At 354, one or more feature vectors are extracted. The one or more feature vectors are provided to an inference model. At 356, the inference model generates one or more predictions associated with the input audio signal. Note that, as shown in FIG. 3B, the input audio signal is a time-series signal that can last for any suitable period of time, e.g., 1 second, 10 seconds, 1 minute, 10 minutes, 30 minutes, 1 hour, 10 hours, etc. Predictions may be generated sequentially for each frame of the input audio signal. In FIG. 3B, the time-series stream of predictions is represented as L1, L2, L3, etc., where L n represents the prediction associated with the nth frame.

[0058] The inference model used to generate a prediction can include various blocks or layers. For example, the inference model can include a positional coding layer configured to embed positional information associated with a block of streaming data. For example, the positional coding layer can indicate positional information of a particular word or speech token within a stream of data. As a more specific example, the positional information can indicate an absolute position (e.g., relative to the beginning of the data stream) or a relative position (e.g., relative to the current time). As a more specific example, the positional information can be combined with extracted features to generate input data that is provided to a transformer layer. The transformer layer can include a streaming multi-head attention block, as shown in FIG. 5 and described in connection with below. The output of the transformer layer can then be provided to a linear layer and then to a softmax layer. The linear layer can be configured to perform a linear transformation of the output of the transformer layer. The output of the softmax layer can be a prediction associated with the block of streaming data. In other words, the softmax layer can be configured to assign probabilities to a set of candidate predictions, and the prediction can then be selected based on the probabilities assigned by the softmax layer.

[0059] FIG. 4 is a diagram of an example of an inference model according to some embodiments. As shown, extracted features 402 and position information generated by a position coding layer 404 can be combined at 404 to generate transformer input data. The transformer input data can be provided to an iterative transformer with streaming attention block 406, which can be configured to generate and / or utilize streaming attention vectors. The output of the iterative transformer with streaming attention block 406 can be provided to a linear layer 408, which can be configured to perform a linear transformation on the output of the transformer with streaming attention layer. The output of the linear layer can be provided to a softmax layer 410, which can be configured to determine probabilities associated with a set of candidate predictions. For example, in an example where the inference model is configured to perform speaker detection for a set of N candidate speakers, the softmax layer can assign a probability that a block of streaming audio data is associated with each of speakers 1, ..., N. The prediction block 412 can then be configured to select a prediction from the set of candidate predictions based on the probabilities assigned by the softmax layer 410.

[0060] FIG. 5 illustrates an example of a transformer layer of an inference model according to some implementations. As shown, inputs may be provided to a layer normalization block 502. Note that the layer normalization block 502 is optional and may be omitted. The output of the layer normalization block 502 (or the input, if the layer normalization block 502 is omitted) may be provided to a streaming multihead attention block 504. The streaming multihead attention block 504 may be configured to generate attention vectors based on blocks of streaming data. In particular, as described in more detail below in conjunction with FIGS. 6 and 9, the streaming multihead attention block 504 may generate attention vectors based on transformed values ​​of queries, keys, and values ​​obtained from corresponding query, key, and value buffers. Each buffer may be a circular buffer configured to update and / or replace values ​​based on time-series streaming data. This allows attention vectors to be generated based on recent streaming data. Examples of the streaming multi-head attention block are shown in Figure 6 (e.g., configured to implement the frame inference method of streaming attention) and Figure 9 (e.g., configured to implement the block inference method of streaming attention) and described below. The output of the streaming multi-head attention block 504 may be provided to a position-wise feedforward layer 506, which may be configured to generate a transformer layer output, which may then be provided to a linear layer, as shown in Figure 4 and described in connection with above.

[0061] In some embodiments, the multi-head streaming attention block may include a query buffer, a key buffer, and a value buffer. Each of the query buffer, key buffer, and value buffer may be configured to store values ​​associated with a frame of streaming data, such that the query buffer, key buffer, and value buffer are updated based on time-series frames. For example, for a given frame of streaming data, a query transform, a key transform, and a value transform may be identified based on input data for the frame of streaming data. Note that in some embodiments, any of the query transform, key transform, and value transform may be a matrix. The query buffer, key buffer, and value buffer may be updated based on the identified query, key, and value transform. For example, a transform associated with the current frame may be added to the buffer, and an old transform may be removed from the buffer. In some embodiments, each buffer may be a circular buffer. In some embodiments, one or more query frames may be read from the updated query buffer. A dot product may be determined based on the frame in the key buffer and the one or more read query frames to determine a weight set. The weight set may be utilized to determine a streaming attention vector.

[0062] FIG. 6 illustrates an example implementation of a multi-head streaming attention block according to some embodiments. The multi-head streaming attention block illustrated in FIG. 6 can be used to implement a streaming attention frame inference method. As illustrated, an input 601 is provided. The input 601 can correspond to a frame of streaming data (e.g., a frame of streaming audio data). The input 601 is provided to a query projection 602, a key projection 603, and a value projection 604. Each of the query projection 602, the key projection 603, and the value projection 604 can generate a linear projection of the input 601 onto the query, key, and value spaces, respectively. The projections from the query projection 602, the key projection 603, and the value projection 604 can be provided to a multi-head attention partition 615. Traditionally, dot multiplications are determined between the query value and every key frame. However, to perform streaming inference, the dot multiplications are determined using the delayed query and key. The delay of the query value is performed by a first-in, first-out (FIFO) delay management block 606, and the delay of the key value is performed by a history delay management block 607. Note that an example implementation of the FIFO delay management block 606 is shown in FIG. 7 and described below, and an example implementation of the history delay management block 607 is shown in FIG. 8 and described below. The dot-product multiplication of the delayed query transformation and the delayed key transformation is performed by a dot-product multiplication block 609. The output of the dot-product multiplication block 609 is a weight set. A delayed version of the value transformation is generated by a history delay management block 608. A weighted sum is generated by a weighted sum block 610 using the delayed value transformation and the weight set. Note that block 605 may be repeated for each head of a multi-head streaming attention block. The outputs of each head are concatenated by a concatenation block 611. A delayed version of the input 601 is generated by a FIFO delay management block 612, and the output of the delayed input and the concatenated sum are provided to an adding residual block 613. The outputs 614 correspond to the outputs of the different heads of the multi-head attention layer with delayed versions of the residuals.

[0063] A FIFO delay manager can be used to delay query translation and input. The FIFO delay manager can be configured to add a current frame representation (e.g., the current input frame, as in the case of FIFO delay management block 612 of FIG. 6, or the current query projection, as in the case of FIFO delay management block 606 of FIG. 6) to the top of a stack and discard the last frame on the stack. This allows the FIFO delay manager to operate as a queue. In some embodiments, the queue may be configured as a circular buffer so that older frames are removed and / or rewritten.

[0064] FIG. 7 illustrates an example of a FIFO delay management block 700, according to an embodiment. In some embodiments, the FIFO delay management block 700 may be implemented as the FIFO delay management blocks 606 and / or 612. As shown, an input frame 701 may be added to the front of a delay line 702. In some embodiments, the delay line 702 may operate with a fixed amount of memory (e.g., if implemented using a circular buffer and / or queue) such that older frames (e.g., older frame 703) are discarded when new frames (e.g., input frame 701) are added to the front of the delay line 702. The output frame 504 may be determined based on a query delay value. In some embodiments, the output frame 504 may be the same as the discarded frame 503 to minimize the amount of memory occupied by the delay line 502.

[0065] As shown in and described above in connection with Figure 6, the history delay management block may be configured to generate delayed versions of value and key transformations. Unlike the FIFO delay management block, the history delay management block may be configured to discard the oldest frame and output the entire remaining stack rather than just one selected frame.

[0066] FIG. 8 shows an example of a history delay management block 800 according to some embodiments. In some embodiments, the history delay management block 800 may be implemented as the history delay management blocks 607 and / or 608 shown in and described above in connection with FIG. 6. As shown, an input frame 801 may be added in front of a delay line 802. The delay line 802 may be implemented as a circular buffer and / or queue. To operate the delay line with a fixed amount of memory, the oldest frame 803 is discarded as the input frame 801 is added in front of the delay line 802. Note that the output of the history delay management block 800 includes all frames of the delay 802 after the input frame has been added and the oldest frame 803 has been discarded.

[0067] In the multi-head attention layer shown in and described above in connection with FIG. 6, the current query block is dot-multiplied with all key blocks of a given frame. However, for networks with multiple layers, this can lead to longer latency and can cause problems when generating predictions related to streaming data. For example, for a six-layer Transformer network operating on two audio frames (e.g., one look-ahead frame and one current frame), given a frame duration of 20 milliseconds, the total latency for the six layers could be 2 * 20 milliseconds * 6 layers, or 240 milliseconds. Here, we disclose a low-latency implementation of a multi-head streaming attention block configured to generate streaming attention vectors using a block inference method. Rather than multiplying the query block of the current frame by every key block, multiple query blocks (e.g., the current query block and N look-ahead blocks, where N is 2, 3, etc.) can be multiplied by every key block. The number of look-ahead blocks N may be the length of the FIFO delay management block (e.g., the length of the delay line). Each successive layer of the transformer can be provided with input from the previous layer, where all future blocks can be replaced with new versions of the look-ahead blocks generated from the previous layer. Replacing future blocks based on the output of the previous layer can prevent delay accumulation. In the above example of audio frames of 20 ms duration and a six-layer transformer, a multi-head streaming attention block that considers two frames, the current frame and the look-ahead frame, can have a delay of 2*20 ms, or 40 ms, rather than the 240 ms associated with using the multi-head attention layer shown in and described above in connection with FIG. 6.

[0068] Figure 9 illustrates an example implementation of a low-latency multi-head streaming attention layer that can be used to implement a block inference method, according to some embodiments. Many components of the low-latency multi-head attention layer illustrated in Figure 9 are similar to the corresponding components of the multi-head attention layer illustrated in Figure 6. However, unlike those illustrated in Figure 6, the FIFO delay management block and history delay management block are implemented as block delay blocks. For example, FIFO delay management blocks 606 and 612 are replaced by block FIFO delay management blocks 906 and 912, and history delay management blocks 607 and 608 are replaced by block history delay management blocks 907 and 908.

[0069] A block FIFO delay management block (e.g., block FIFO delay management blocks 906 and 912) may be used to identify delay queries and input blocks. A first version of the block FIFO delay management block may be used for the first layer of the network, where an input frame is added to a delay line and the oldest frame is discarded. The output for the first frame may be multiple frames, e.g., the current frame and N look-ahead frames, where N is 1, 2, 3, 4 frames, etc. A second version of the block FIFO delay management block may be used for subsequent layers of the network, where the input to the delay line is the same as the output. Propagating the input to the output may prevent delay from accumulating throughout successive layers of the transformer network.

[0070] FIG. 10A illustrates an example of a block FIFO delay management block that may be used in connection with the first layer of a network, according to some embodiments. As shown, a current frame 1001 is applied to a delay line 1002. The delay line 1002 may be implemented as a circular buffer, a queue, or the like. The oldest frame 1003 is discarded from the end of the delay line 1002. The output 1004 is all frames in the delay line after discarding the oldest frame 1003. FIG. 10B illustrates an example of a block FIFO delay management block that may be used in connection with a subsequent layer of a network, according to some embodiments. As shown, the output 1054 is the same as the input 1051.

[0071] 10A and 10B includes a current frame and two look-ahead frames, it should be understood that this is merely an example. In some embodiments, the delay line may be configured to operate with a current frame and N look-ahead frames, where N is 1, 2, 4, 5, etc. Nevertheless, for low-latency implementations of the multi-head streaming attention block, the block FIFO delay management block may be configured to output the current frame plus N look-ahead frames.

[0072] The block history delay management block can be used to maintain buffers for key and value transformations in the low-latency multi-head attention layer. The block history delay management block can be configured to add and / or replace a set of frames in the delay line (e.g., a current frame and N look-ahead frames) and discard the oldest frame from the delay line. For example, a current frame can be added to the delay line, and N look-ahead frames can replace N frames in the delay line. To operate on fixed memory, the oldest frame can be discarded from the delay line.

[0073] FIG. 11 shows an example of a block history delay management block that can be used in some embodiments of a low-latency multi-head streaming attention block according to some embodiments. As shown, a set of frames 1101 can include a current frame and N look-ahead frames. In the example shown in FIG. 11, N is 2, and the set of frames 1101 includes a total of three frames. The current frame is added to a delay line 1102, and the two look-ahead frames replace two frames in the delay line 1102. The oldest frame 1103 is discarded. After updating the delay line (e.g., a circular buffer or queue), the key and / or value transformations shown in the updated delay line can be used to generate a weight set and / or weighted sum, as shown in and described above in connection with FIG. 9.

[0074] In some embodiments, attention vectors can be generated for each frame of streaming data. For example, during inference, attention vectors can be generated based on a window of input frames rather than every input frame (e.g., as conventionally used to generate attention vectors for non-streaming data). In some embodiments, attention vectors can be generated by a multi-head streaming attention block that manipulates time delays, generally referred to herein as a “frame inference method.” For example, as shown in FIG. 6, such an implementation can utilize a FIFO delay management block to delay query translation frames and a history delay management block to delay key-value translation frames. The multi-head streaming attention block can then be stacked at each layer to generate attention vectors. As another example, as shown in FIG. 9, attention vectors can be generated by a multi-head streaming attention block that reduces latency by preventing latency from stacking across multiple layers, generally referred to herein as a “block inference method.” As shown in FIG. 9 and described above in connection with the block inference method, a look-ahead frame can be inferred for each layer to prevent latency accumulation.

[0075] Regardless of whether the frame inference method or the block inference method is used, the attention vector can be generated for each frame by updating buffers associated with a query, key, and value transformation (e.g., a matrix). For example, for each frame of streaming data, a query, key, and value transformation can be identified based on input data associated with the frame. The query, key, and value buffers can be updated based on the query, key, and value transformation, such that the query, key, and value buffers can be configured to store values ​​associated with a previous frame of streaming data. After updating the buffers, one or more query frames can be retrieved from the updated query buffer to process the input data. For example, in a frame inference method, one query frame can be used to process the input data, while in a block inference method, multiple query frames can be used (e.g., corresponding to a current query frame and one or more look-ahead frames). A dot product can be determined for a frame in the key buffer and one or more query frames to determine a weight set. A weighted sum can then be determined using the weight set and the frames in the value buffer. The weighted sum can then be utilized to determine a streaming attention vector, for example, by applying a softmax function to the weighted sum. The streaming attention vector can be used to generate predictions related to streaming data.

[0076] 12 is a flowchart illustrating an example process 1200 for generating a streaming attention vector. As illustrated, the blocks of process 1200 may be performed by one or more control systems and / or processors, such as, for example, a server device, a video or audio conferencing device, a laptop device, etc. In some embodiments, the blocks of process 1200 may be performed in a different order than that shown in FIG. 12. In some embodiments, one or more blocks of process 1200 may be performed in a different order than that shown in FIG. 12. In some implementations, one or more blocks of method 1200 may be omitted.

[0077] Process 1200 may begin, at 1202, by obtaining input data representing a frame of streaming data. The frame of streaming data may be obtained by a laptop device, a desktop computing device, an audio or video conferencing device, etc. The streaming data may include streaming audio data obtained by one or more microphones. The streaming data may include speech or other audio data for which prediction is performed by a network, e.g., a transformer network. Note that the input data representing the frame of streaming data may include one or more extracted features associated with the frame, a positional coding associated with the frame, etc.

[0078] At 1204, the process 1200 can identify query, key, and value transformations based on input data for the frame of streaming data. The query, key, and value transformations, in some embodiments, can be in the form of query, key, and value matrices. In some implementations, the query, key, and value transformations can be identified using techniques implemented by a non-streaming network, such as a non-streaming transformer network.

[0079] At 1206, process 1200 can update the query buffer, key buffer, and value buffer based on the query, key, and value transformations. The query, key, and value buffers may each be configured to store parameters associated with a previous frame of streaming data. For example, in a frame inference method, as shown and described above in connection with Figures 6, 7, and 8, the buffers may each be updated based on parameters associated with the current frame. As another example, in a block inference method, as described above in connection with Figures 9, 10, and 11, the buffers may each be updated based on parameters associated with the current frame and one or more look-ahead frames.

[0080] At 1208, process 1200 can read one or more query frames from the updated query buffer to be used to process the input data. For example, a frame inference method can be used to read a single query frame, as shown in Figure 7 and described above. As another example, a block inference method can be used to read multiple query frames, as shown in Figures 10A and 10B and described above.

[0081] At 1210, a dot product of the retrieved query frame and the key frames in the key buffer can be determined to generate a weight set. In a frame inference method, the dot product can include a dot product of a single retrieved query frame with a key frame in the key buffer. Using a block inference method, the dot product can include dot products of multiple retrieved query frames with key frames in the key buffer.

[0082] At 1212, a weighted sum may be determined between the weight set determined in block 1210 and the frame in the value buffer. The weighted sum may correspond to an unnormalized attention vector.

[0083] Note that blocks 1208-1212 may be repeated for each layer of the network. For example, in the case of a six-layer Transformer network, blocks 1208-1212 may be repeated for each of the six layers. Each of blocks 1208-1212 may be performed by a multi-head streaming attention block (e.g., as shown in and described above in connection with FIG. 6 and / or FIG. 9), where each layer is associated with a different head.

[0084] The weighted sum may be used to generate a streaming attention vector at 1214. For example, the unnormalized streaming attention vector generated at block 1212 may be normalized by applying the unnormalized streaming attention vector to a linear layer and / or a softmax layer, as shown in and described above in connection with FIG. 4. The output of the softmax layer may be a normalized streaming attention vector that may be used to generate one or more predictions associated with a frame of streaming data.

[0085] In some embodiments, a network utilizing streaming attention (e.g., by implementing a multi-head streaming attention block as shown in and described above in conjunction with FIGS. 6, 9, and 12) may be trained using a multi-stage training process. For example, the network may be trained by first training a version of the network utilizing non-streaming attention. As a more specific example, such a network may utilize a non-streaming multi-head attention block that does not use query, key, and value buffers. In such a non-streaming network, attention vectors may be generated using every frame of data rather than a window of frames. Weights in such a network may be updated based on backpropagation and / or other suitable training techniques. FIG. 13 is an example flowchart for performing pre-training on a streaming attention network.

[0086] After pre-training a version of the network without streaming attention, the model can be fine-tuned using a fine-tuning phase of training. Fine-tuning the network may be performed using different techniques. For example, as shown in FIG. 14A and described in conjunction with below, fine-tuning can be performed by using backpropagation to update weights associated with streaming attention kernels for a second version of the network that includes query, key, and value buffers. As another example, as shown in FIG. 14B and described in conjunction with below, fine-tuning can be performed by providing a series of time-shifted segments to a version of the network without streaming attention and updating weights based on an aggregated set of predictions associated with the time-shifted segments.

[0087] 13 is a flowchart of an example process 1300 for pre-training a streaming attention network using a version of the network that does not include a streaming attention block. The blocks of process 1300 may be implemented by one or more control systems and / or processors of a device, such as a server device or other computing device, suitable for training the model. In some embodiments, the blocks of process 1300 may be performed in an order different from the order shown in FIG. 13. In some embodiments, two or more of the blocks of process 1300 may be performed substantially in parallel. In some implementations, one or more blocks of method 1300 may be omitted.

[0088] At 1302, process 1300 may obtain training data. The training data may be the same type of data that a trained network configured to generate predictions for streaming data is configured to obtain as input. For example, if the trained network is configured to operate on streaming audio data, the training data may correspond to frames of audio data. As a more specific example, if the trained network is configured to generate predictions related to speech in the streaming audio data, the training data may include frames of speech audio data.

[0089] At 1304, process 1300 may generate a predicted output using a machine learning model that uses non-streaming attention. For example, the machine learning model may be a conventional Transformer network that does not include a multi-head streaming attention block. As a more specific example, rather than generating an attention vector based on a window of query frames, the machine learning model may generate an attention vector based on all query frames.

[0090] At 1306, process 1300 may determine a loss based on the training data and the predicted output. For example, the training data may include corresponding ground truth output labels, and the loss may be determined based on the difference between the predicted output and the ground truth output labels.

[0091] At 1308, the process 1300 can update weights associated with the model using backpropagation. Blocks 1302 through 1308 can be repeated until a stopping criterion for pre-training the network is reached. For example, the stopping criterion can include a loss being below a predetermined threshold, weights changing by less than a predetermined threshold after each iteration, etc.

[0092] FIG. 14A is a flowchart of an example process 1400 for fine-tuning a model configured to operate on streaming data to generate streaming attention vectors after a pre-training process for the model is completed (e.g., the pre-training process shown in FIG. 13 and described above in connection therewith). In some embodiments, the blocks of process 1400 may be performed by, for example, one or more control systems or processors of a server device or other computing device suitable for training a model. In some embodiments, the computing device that performs process 1400 may be the same as or a different computing device from the computing device that performs process 1300. In some embodiments, the blocks of process 1400 may be performed in a different order than the order shown in FIG. 14A. In some embodiments, two or more of the blocks of process 1400 may be performed substantially in parallel. In some implementations, one or more blocks of method 1400 may be omitted.

[0093] At 1402, process 1400 may obtain training data. The training data may be the same type of data that a trained network configured to generate predictions for streaming data is configured to obtain as input. For example, if the trained network is configured to operate on streaming audio data, the training data may correspond to blocks of audio data. As a more specific example, if the trained network is configured to generate predictions related to speech in the streaming audio data, the training data may include blocks of speech audio data.

[0094] At 1404, process 1400 may generate a predicted output using a machine learning model that uses streaming attention. For example, the machine learning model may have a multi-headed streaming attention block, as shown in and described above in connection with Figures 6 and 9. The machine learning model may be configured to generate the predicted output by generating a streaming attention vector using a window of frames, rather than all previous frames.

[0095] At 1406, process 1400 may determine a loss based on the training data and the predicted output. For example, the training data may include corresponding ground truth output labels, and the loss may be determined based on the difference between the predicted output and the ground truth output labels.

[0096] At 1408, the process 1400 can update weights associated with the model using backpropagation with derivatives derived from the streaming attention kernel. In some embodiments, the forward portion and derivatives for training can be determined from the derived equations. The forward portion and derivatives can be implemented using a multi-threaded framework (e.g., using the Compute Unified Device Architecture (CUDA) framework, or other similar multi-threaded framework). Note that the implementation can be directly invoked from a toolkit such as PyTorch.

[0097] In some embodiments, fine-tuning of a pre-trained model can be achieved by providing time-shifted input segments to the model and aggregating the model's outputs in response to the time-shifted input segments. Note that the model may not include a component configured to adapt to streaming inputs, such as a multi-head streaming attention block. Furthermore, note that the techniques described herein for fine-tuning a pre-trained model using time-shifted input segments can be applied to fine-tuning any suitable model architecture (e.g., a transformer network, a conformer network, etc.) to adapt the model to utilize streaming data as input. In some embodiments, fine-tuning can be performed by providing a series of time-shifted segments to the model for each input training sample to generate a corresponding series of predicted outputs. The series of predicted outputs can then be aggregated. A loss can be determined based on the aggregated series of predicted outputs. The weights of the model (e.g., not including a streaming attention component) can then be updated based on the loss. Note that the input data can include a kernel having a number of frames, where the number of frames includes a number of lookback frames, a current frame, and a number of lookahead frames. The time shifting of the segments can depend on a hop size parameter. In some embodiments, the hop size can correspond to a frame or block size. By time shifting the segments and providing them as separate inputs to the model, mixing of information from frame to frame can be avoided. Note that for block inference methods, multiple outputs can be passed from layer to layer, each corresponding to a different latency from one layer to another, and the final layer can produce a single output from which the loss can be determined. The final layer can select the frame that corresponds to the current time. For frame inference methods, each layer can output a single frame, such that the final layer similarly outputs a single frame.Note that the outputs of different layers may be aggregated (e.g., using a weighted sum) before performing other operations such as downstream tasks by different layers, or before utilizing different loss functions.

[0098] 14B is a flowchart of an example process 1450 for performing model fine-tuning for performing streaming attention inference by time-shifting input segments according to some embodiments. In some embodiments, the blocks of process 1450 may be performed by, for example, one or more control systems or processors of a server device or other computing device suitable for training a model. In some embodiments, the computing device performing process 1450 may be the same as or a different computing device from the computing device performing process 1300. In some embodiments, the blocks of process 1450 may be performed in an order different from the order shown in FIG. 14B. In some embodiments, two or more of the blocks of process 1450 may be performed substantially in parallel. In some implementations, one or more blocks of method 1450 may be omitted.

[0099] Process 1450 may begin, at 1452, by obtaining training data. The training data may be the same type of data that the trained network, configured to generate predictions for streaming data, is configured to obtain as input. For example, if the trained network is configured to operate on streaming audio data, the training data may correspond to blocks of audio data. As a more specific example, if the trained network is configured to generate predictions related to speech in the streaming audio data, the training data may include blocks of speech audio data.

[0100] At 1454, process 1450 may generate, for a vector of data included in the training data that includes a lookback block (e.g., a previous frame of data), a current block (e.g., a current frame of audio data), and a lookahead block (e.g., a future frame), a series of segments including blocks shifted in time by a period corresponding to the current block. Note that if the data corresponds to audio data, the period corresponding to the current block may be a frame period (e.g., 10 ms, 12 ms, 20 ms, etc.). The vector of data is generally referred to herein as a kernel, and the period of the shift is generally referred to herein as a hop size.

[0101] Referring to Figure 15, there is shown an input block of data 1502. From the input block of data 1502, vectors of data corresponding to one or more lookback blocks, a current block, and one or more lookahead blocks may be identified (e.g., as part of the input block of data 1502). Panel 1504 shows a series of segments that may be generated based on the vectors of data as shown in Figure 15, with each segment time shifted by a period corresponding to the period of the current block.

[0102] Returning to Figure 14B, at 1456, process 1450 can generate a set of predicted outputs by providing each segment to a model. Note that, as discussed above, the model may not incorporate components for processing streaming data, such as a multi-head streaming attention block. For example, the model may be a traditional transformer network. One output may be generated for each input segment.

[0103] Referring to FIG. 15, panel 1506 shows that a series of segments are arranged into batches before being provided to the model.

[0104] 14B, the process 1450 may aggregate the set of predicted outputs at 1458. For example, aggregating the set of predicted outputs may include applying different loss functions.

[0105] In FIG. 15, panel 1508 shows a set of aggregated prediction outputs that are aggregated based on the time shift of the corresponding time-shifted segments in panel 1504, according to some embodiments.

[0106] 14B, at 1460, the process 1400 can determine a loss based on the aggregated set of predicted outputs. For example, the loss can be determined based on the difference between the aggregated set of predicted outputs and a ground truth prediction associated with the training data.

[0107] At 1462, process 1450 can update weights associated with the model using backpropagated derivatives derived from a model that does not include a component for streaming attention and based on the loss determined at 1460.

[0108] After the model has been fine-tuned, various components of the model may be replaced with streaming attention versions. For example, the multi-head attention block of a conventional Transformer network that has been fine-tuned using the technique described in Figure 14B may be replaced with a multi-head streaming attention block, e.g., as shown in and described above in connection with Figures 6 and / or 9.

[0109] FIG. 16 is a block diagram illustrating example components of a device capable of implementing various aspects of the present disclosure. As with other figures provided herein, the number and types of elements shown in FIG. 16 are merely examples. Other implementations may include more, fewer, and / or different types and numbers of elements. According to some examples, device 1600 may be configured to perform at least some of the methods disclosed herein. In some embodiments, device 1600 may be or include a television, one or more components of an audio system, a mobile device (such as a mobile phone), a laptop computer, a tablet device, a smart speaker, or another type of device.

[0110] In some alternative implementations, device 1600 may be or include a server. In some such implementations, device 1600 may be or include an encoder. Thus, in some examples, device 1600 may be a device configured for use in an audio environment, such as a home audio environment, while in other examples, device 1600 may be a device configured for use in the "cloud," such as a server.

[0111] In this example, device 1600 includes interface system 1605 and control system 1610. Interface system 1605, in some implementations, may be configured to communicate with one or more other devices in an audio environment. The audio environment, in some examples, may be a home audio environment. In other examples, the audio environment may be another type of environment, such as an office environment, an automobile environment, a train environment, a street or sidewalk environment, a park environment, etc. Interface system 1605, in some implementations, may be configured to exchange control information and associated data with audio devices in the audio environment. The control information and associated data, in some examples, may be related to one or more software applications running on device 1600.

[0112] The interface system 1605, in some implementations, may be configured to receive or provide a content stream. The content stream may include audio data. The audio data may include, but is not limited to, an audio signal. In some cases, the audio data may include spatial data, such as channel data and / or spatial metadata. In some examples, the content stream may include video data and audio data corresponding to the video data.

[0113] The interface system 1605 may include one or more network interfaces and / or one or more external device interfaces (e.g., one or more universal serial bus (USB) interfaces). According to some implementations, the interface system 1605 may include one or more wireless interfaces. The interface system 1605 may include one or more devices implementing a user interface, such as one or more microphones, one or more speakers, a display system, a touch sensor system, and / or a gesture sensor system. In some examples, the interface system 1605 may include one or more interfaces between the control system 1610 and a memory system, such as the optional memory system 1615 shown in FIG. 16 . However, in some cases, the control system 1610 may include a memory system. In some implementations, the interface system 1605 may be configured to receive input from one or more microphones in the environment.

[0114] The control system 1610 may include, for example, a general-purpose single or multi-chip processor, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), or other programmable logic elements, discrete gate or transistor logic, and / or discrete hardware components.

[0115] In some implementations, the functionality of the control system 1610 may reside on more than one device. For example, a portion of the control system 1610 may reside on a device within one of the environments described herein, while another portion of the control system 1610 may reside on a device external to the environment, such as a server, a mobile device (e.g., a smartphone, or a tablet computer), etc. In other examples, a portion of the control system 1610 may reside on a device within one environment, while another portion of the control system 1610 may reside on one or more other devices of the environment. For example, a portion of the control system 1610 may reside on a device implementing a cloud-based service, such as a server, while another portion of the control system 1610 may reside on another device implementing the cloud-based service, such as another server or memory device. The interface system 1605 may also reside on multiple devices in some examples. In some examples, a portion of the control system may reside within or on the earphones.

[0116] In some implementations, the control system 1610 may be configured to perform, at least in part, the methods disclosed herein. According to some embodiments, the control system 1610 may be configured to perform methods such as updating query, key, and value buffers, generating streaming attention vectors based on buffered query, key, and value parameters, etc.

[0117] Some or all of the methods described herein may be performed by one or more devices according to instructions (e.g., software) stored on one or more non-transitory media. Such non-transitory media may include memory devices as described herein, including, but not limited to, random access memory (RAM) devices, read-only memory (ROM) devices, etc. One or more non-transitory media may reside, for example, in optional memory system 1615 shown in FIG. 1 and / or in control system 1610. Thus, various novel aspects of the subject matter described in this disclosure may be implemented in one or more non-transitory media having software stored thereon. The software may, for example, extract objects from a multi-channel audio signal, generate a spatial enhancement mask, apply the spatial enhancement mask, generate an output binaural audio signal, etc. The software may be executable by one or more components of a control system, such as control system 1610 of FIG. 16.

[0118] In some examples, device 1600 may include optional microphone system 1620 shown in FIG. 16 . Optional microphone system 1620 may include one or more microphones. In some implementations, one or more microphones may be part of or associated with another device, such as a speaker of a speaker system, a smart audio device, or the like. In some examples, device 1600 may not include microphone system 1620. However, in some such implementations, device 1600 may nevertheless be configured to receive microphone data of one or more microphones in the audio environment via interface system 1610. In some such implementations, a cloud-based implementation of device 1600 may be configured to receive microphone data, or noise metrics corresponding at least in part to the microphone data, from one or more microphones in the audio environment via interface system 1610.

[0119] According to some implementations, device 1600 may include an optional speaker system 1625 shown in FIG. 16. Optional speaker system 1625 may include one or more speakers and may be referred to herein as a "speaker" or more generally as an "audio reproduction transducer." In some instances (e.g., cloud-based implementations), device 1600 may not include speaker system 1625. In some implementations, device 1600 may include headphones. The headphones may be connected or coupled to device 1600 via a headphone jack or a wireless connection (e.g., BLUETOOTH).

[0120] Some aspects of the present disclosure include systems or devices configured (e.g., programmed) to perform one or more examples of the disclosed methods, and tangible computer-readable media (e.g., disks) storing code for implementing one or more examples of the disclosed methods or steps thereof. For example, some disclosed systems may be or include a programmable general-purpose processor, digital signal processor, or microprocessor programmed with software or firmware and / or otherwise configured to perform any of various operations on data, including embodiments of the disclosed methods or steps thereof. Such a general-purpose processor may be or include a computer system including input devices, memory, and processing subsystems programmed (and / or otherwise configured) to perform one or more examples of the disclosed methods (or steps thereof) in response to data asserted thereto.

[0121] Some embodiments can be implemented as a configurable (e.g., programmable) digital signal processor (DSP) configured (e.g., programmed and otherwise configured) to perform necessary processing on an audio signal, including performing one or more examples of the disclosed methods. Alternatively, embodiments of the disclosed system (or elements thereof) can be implemented as a general-purpose processor (e.g., a personal computer (PC), other computer system, or microprocessor (which may include input devices and memory)) that is programmed with software or firmware and / or otherwise configured to perform any of a variety of operations, including one or more examples of the disclosed methods. Alternatively, elements of some embodiments of the inventive system are implemented as a general-purpose processor or DSP configured (e.g., programmed) to perform one or more examples of the disclosed methods, and the system also includes other elements (e.g., one or more speakers and / or one or more microphones). A general-purpose processor configured to perform one or more examples of the disclosed methods can be coupled to an input device (e.g., a mouse and / or keyboard), memory, and a display device.

[0122] Another aspect of the present disclosure is a computer-readable medium (e.g., a disk or other tangible storage medium) that stores code for performing (e.g., a coder executable file for performing) one or more examples of the disclosed methods or steps thereof.

[0123] While particular embodiments of the disclosed invention and applications of the disclosure have been described herein, it will be apparent to those skilled in the art that many variations to the embodiments and applications described herein are possible without departing from the scope of the disclosure as described and claimed herein. It should be understood that, while particular forms of the disclosure have been shown and described, the disclosure is not limited to the specific embodiments described and shown or to the particular methods described.

Claims

1. 1. A method for processing streaming data, the method comprising: obtaining input data representing frames of streaming data; identifying a query transformation, a key transformation, and a value transformation based on the input data of the frame of streaming data; updating a query buffer, a key buffer, and a value buffer based on the identified query, key, and value transformations, such that the query buffer, the key buffer, and the value buffer are configured to store parameters associated with a previous frame of streaming data and the frame of streaming data, respectively; reading one or more query frames from the updated query buffer for use in processing the input data; determining a dot product of the one or more retrieved query frames and a frame in the key buffer to determine a set of weights; determining a weighted sum between the set of weights and a frame in the value buffer; utilizing the weighted sum to generate a streaming attention vector, the streaming attention vector usable by a network to generate predictions related to the streaming data; A method comprising:

2. The method of claim 1 , wherein the network is a transformer network.

3. The method according to claim 1 or 2, wherein the streaming data is streaming audio data.

4. The method of any one of claims 1 to 3, wherein the prediction related to the streaming data comprises a prediction of an audio emotion related to the streaming data.

5. 5. The method of claim 1, wherein the predictions related to the streaming data include identifying one or more features useful for providing to one or more downstream machine learning models.

6. The method of claim 5 , wherein the one or more features include an identification of one or more speakers associated with the streaming data.

7. The method of any one of claims 1 to 6, wherein the predictions associated with the streaming data include classifications of one or more words or phonemes of the streaming data.

8. The method of any one of claims 1 to 7, wherein at least one of the query buffer, the key buffer, or the value buffer is a circular buffer.

9. The step of updating the query buffer comprises: adding a current query frame based on the query transformation to the query buffer; discarding the oldest query frame in the query buffer; The method according to any one of claims 1 to 8, comprising:

10. The method of claim 9 , wherein the one or more query frames retrieved correspond to the oldest query frame discarded.

11. The network includes a plurality of layers, and the step of updating the query buffer includes, for a first layer of the plurality of layers: adding a current query frame based on the query transformation to the query buffer; replacing a plurality of query frames in the query buffer with a plurality of look-ahead query frames corresponding to future times; discarding the oldest query frame in the query buffer; The method according to any one of claims 1 to 8, comprising:

12. 12. The method of claim 11, wherein the one or more query frames read for use in processing the input block by the first layer of the plurality of layers include the current query frame and the plurality of look-ahead query frames.

13. The method of claim 11 or 12, wherein the plurality of look-ahead query frames comprises two look-ahead frames.

14. 14. The method of claim 11, wherein the one or more query frames read out for use in processing the input block by each of the plurality of layers other than the first layer are passed to a given layer by a previous layer.

15. Updating the key buffer and updating the value buffer comprises: adding a current key frame to the key buffer based on the key transformation and discarding the oldest key frame from the key buffer; adding a current value frame to the value buffer based on the value transformation; discarding the oldest value frame from the value buffer; The method according to any one of claims 1 to 14, comprising:

16. Updating the key buffer and updating the value buffer comprises: adding a current key frame to the key buffer; replacing a plurality of key frames in the key buffer with a plurality of look-ahead key frames based on the key transformation; discarding the oldest key frame from the key buffer; adding a current value frame to the value buffer; replacing a plurality of value frames in the value buffer with a plurality of look-ahead value frames based on the value transformation; discarding the oldest value frame from the value buffer; The method according to any one of claims 1 to 14, comprising:

17. The network comprises: performing initial training using a first version of the network that does not utilize the query buffer, the key buffer, and the value buffer to store a subset of query frames, key frames, and value frames; performing subsequent training that modifies weights associated with the network; The method according to any one of claims 1 to 16, wherein the method is trained by

18. 18. The method of claim 17, wherein the subsequent training is performed using a second version of the network that includes the query buffer, the key buffer, and the value buffer, and performing the subsequent training includes performing backpropagation using derivatives derived from the second version of the network.

19. The subsequent training is performed using the first version of the network, and performing the subsequent training includes: providing the first version of the network with a given training data block of data, a series of time-shifted segments, to generate a corresponding series of predicted outputs; aggregating the set of predicted outputs; determining a loss based on the aggregated set of predicted outputs; and updating weights associated with the first version of the network based on the losses; and 18. The method of claim 17, comprising:

20. 1. A system comprising: one or more processors; a non-transitory computer readable storage medium storing instructions that, when executed by the one or more processors, cause the one or more processors to perform the method of any one of claims 1 to 19; and A system including:

21. 20. A non-transitory computer readable medium storing instructions that, when executed by the one or more processors, cause the one or more processors to perform the method of any one of claims 1 to 19.

Images