Processing streaming data

Abstract

Techniques for processing streaming data are disclosed herein. In some embodiments, the technique includes obtaining input data representative of a streaming data frame. The technique may include identifying a query transform, a key transform, and a value transform based on input data. The technique may include updating a query buffer, a key buffer, and a value buffer such that each buffer is configured to store a parameter associated with a previous streaming data frame and the streaming data frame. The technique may include retrieving one or more query frames from a query buffer. The technique may include determining a dot product of a query frame and a frame in a key buffer to determine a set of weights. The technique may include determining a weighted sum between the frame in the value buffer and the set of weights, and generating a streaming attention vector using the weighted sum.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to U.S. Provisional Application No. 63 / 511,799 filed on July 3, 2023, U.S. Provisional Application No. 63 / 416,429 filed on October 14, 2022, and U.S. Provisional Application No. 63 / 401,046 filed on August 25, 2022, all of which are incorporated herein by reference in their entirety. Technical Field

[0003] The present disclosure relates to systems, methods, and media for processing streaming data. Background Art

[0004] Attention, such as implemented as attention vectors, has become an important component of machine learning algorithms and neural network architectures, bringing advances to areas such as natural language processing, computer vision, speech, and audio processing. For example, attention mechanisms have been implemented in transformer neural networks, which can be used for natural language processing, such as translating text from one language to another, outputting persuasive text blocks in response to queries, and so on. However, applying these algorithmic and neural network advances to streaming data, such as streaming audio data, is difficult due to the limited amount of data available in streaming environments.

[0005] Symbols and terminology

[0006] Throughout this disclosure, including in the claims, the terms "speaker," "loudspeaker," and "audio reproduction transducer" are used synonymously to refer to any sound-producing transducer (or set of transducers). A typical set of headphones includes two speakers. Speakers may be implemented to include multiple transducers (e.g., a woofer and a tweeter), which may be driven by a single common speaker feed or multiple speaker feeds. In some examples, the speaker feeds may undergo different processing in different circuit branches coupled to different transducers.

[0007] Throughout this disclosure, including in the claims, the expression "performing an operation on" a signal or data (e.g., filtering, scaling, transforming, or applying a gain to the signal or data) is used broadly to mean performing the operation directly on the signal or data, or performing the operation on a processed version of the signal or data (e.g., performing the operation on a version of the signal that has been initially filtered or preprocessed before performing the operation).

[0008] Throughout this disclosure, including in the claims, the expression "system" is used in a broad sense to refer to a device, system, or subsystem. For example, a subsystem that implements a decoder may be referred to as a decoder system, and a system that includes such a subsystem (e.g., a system that generates X output signals in response to multiple inputs, where the subsystem generates M inputs and another XM inputs are received from external sources) may also be referred to as a decoder system.

[0009] Throughout this disclosure, including in the claims, the term "processor" is used in a broad sense to refer to a system or device that is programmable or otherwise configurable (e.g., using software or firmware) to perform operations on data (e.g., audio, video, or other image data). Examples of processors include field programmable gate arrays (or other configurable integrated circuits or chipsets), digital signal processors that are programmed and / or otherwise configured to perform pipeline processing of audio or other sound data, programmable general-purpose processors or computers, and programmable microprocessor chips or chipsets. Summary of the invention

[0010] Disclosed herein is a technique for processing streaming data. In some embodiments, a method for processing streaming data includes obtaining input data representing a streaming data frame. The method further includes identifying a query transformation, a key transformation, and a value transformation based on the input data of the streaming data frame. The method further includes updating a query buffer, a key buffer, and a value buffer based on the identified query transformation, key transformation, and value transformation, so that the query buffer, the key buffer, and the value buffer are all configured to store parameters associated with a previous streaming data frame and the streaming data frame. The method further includes retrieving one or more query frames from the updated query buffer for processing the input data. The method further includes determining a dot product of the retrieved one or more query frames with a frame in the key buffer to determine a set of weights. The method further includes determining a weighted sum between the set of weights and the frames in the value buffer. The method further includes generating a streaming attention vector using the weighted sum, wherein the network can use the streaming attention vector to generate predictions associated with the streaming data.

[0011] In some examples, the network is a transformer network.

[0012] In some examples, the streaming data is streaming audio data.

[0013] In some examples, the prediction associated with the streaming data includes a prediction of a speech emotion associated with the streaming data.

[0014] In some examples, the prediction associated with the streaming data includes an identification of one or more features that can be used to provide to one or more downstream machine learning models. In some examples, the one or more features include an identification of one or more speakers associated with the streaming data.

[0015] In some examples, the prediction associated with the streaming data includes a classification of one or more words or phonemes of the streaming data.

[0016] In some examples, at least one of the query buffer, the key buffer, or the value buffer is a circular buffer.

[0017] In some examples, updating the query buffer includes appending a current query frame based on the query transformation to the query buffer; and discarding an oldest query frame in the query buffer. In some examples, the retrieved one or more query frames correspond to the discarded oldest query frame.

[0018] In some examples, the network includes multiple layers, and wherein updating the query buffer includes, for a first layer of the multiple layers: appending a current query frame based on a query transformation to the query buffer; replacing multiple query frames in the query buffer with multiple forward query frames corresponding to future times; and discarding the oldest query frame in the query buffer. In some examples, the one or more query frames retrieved by the first layer of the multiple layers for processing the input block include the current query frame and the multiple forward query frames. In some examples, the multiple forward query frames include two forward frames. In some examples, the one or more query frames retrieved by each layer of the multiple layers except the first layer for processing the input block are passed to the given layer by the previous layer.

[0019] In some examples, updating the key buffer and updating the value buffer include: appending a current key frame based on a key transformation to the key buffer and discarding an oldest key frame from the key buffer; appending a current value frame based on a value transformation to the value buffer; and discarding the oldest value frame from the value buffer.

[0020] In some examples, updating a key buffer and updating a value buffer include: appending a current key frame to the key buffer; replacing multiple key frames in the key buffer with multiple forward-looking key frames based on a key transformation; discarding the oldest key frame from the key buffer; appending a current value frame to the value buffer; replacing multiple value frames in the value buffer with multiple forward-looking value frames based on a value transformation; and discarding the oldest value frame from the value buffer.

[0021] In some examples, the network is trained by performing initial training using a first version of the network that does not utilize a query buffer, a key buffer, and a value buffer to store subsets of query frames, key frames, and value frames; and performing subsequent training that modifies weights associated with the network. In some examples, the subsequent training is performed using a second version of the network that includes a query buffer, a key buffer, and a value buffer, and wherein performing the subsequent training includes performing backpropagation using derivatives derived from the second version of the network. In some examples, the subsequent training is performed using a first version of the network, and wherein performing the subsequent training includes providing the first version of the network with data for a given training data block, a series of time-shifted segments to generate a corresponding series of prediction outputs; aggregating the series of prediction outputs; determining a loss based on the aggregated series of prediction outputs; and updating weights associated with the first version of the network based on the loss.

[0022] Some or all of the operations, functions, and / or methods described herein may be performed by one or more devices according to instructions (e.g., software) stored on one or more non-transitory media. Such non-transitory media may include memory devices such as those described herein, including but not limited to random access memory (RAM) devices, read-only memory (ROM) devices, etc. Therefore, some innovative aspects of the subject matter described in the present disclosure may be implemented by one or more non-transitory media storing software.

[0023] At least some aspects of the present disclosure may be implemented by an apparatus. For example, one or more devices may be capable of at least partially performing the methods disclosed herein. In some implementations, the apparatus is or includes an audio processing system having an interface system and a control system. The control system may include one or more general-purpose single-chip or multi-chip processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field programmable gate arrays (FPGAs) or other programmable logic devices, discrete gate or transistor logic, discrete hardware components, or a combination thereof.

[0024] The details of one or more embodiments of the subject matter described in this specification are set forth in the accompanying drawings and the following description. Other features, aspects, and advantages will become apparent from the description, drawings, and claims. It should be noted that the relative dimensions of the following figures may not be drawn to scale. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 is a diagram illustrating an example audio environment in accordance with some embodiments.

[0026] Figure 2 is a flowchart of an example process for training and using a machine learning model according to some embodiments.

[0027] Figure 3Ais a flow chart of an example process for generating a prediction related to an input signal according to some embodiments.

[0028] Figure 3B is a diagram illustrating the use of extracted features to generate predictions according to some embodiments.

[0029] Figure 4 is a block diagram of a portion of a transformer network utilizing a streaming attention mechanism according to some embodiments.

[0030] Figure 5 Depicted is a streaming attention layer of a transformer network in accordance with some embodiments.

[0031] Figure 6 Depicted are components of a multi-head attention layer according to some embodiments.

[0032] Figure 7 Operation of a first-in, first-out (FIFO) delay manager is depicted in accordance with some embodiments.

[0033] Figure 8 Operation of a history delay manager according to some embodiments is depicted.

[0034] Fig. 9 Depicted are components of a low-latency multi-head attention layer according to some embodiments.

[0035] Fig. 10A and 10B Depicted is the operation of a FIFO delay manager that may be used in conjunction with a low-latency multi-headed attention layer in accordance with some embodiments.

[0036] Fig.11 Depicted is the operation of a history delay manager that may be used in conjunction with a low-latency multi-head attention layer in accordance with some embodiments.

[0037] Fig.12 is a flowchart of an example process for generating a streaming attention vector according to some embodiments.

[0038] Fig.13 is a flow chart of an example process for pre-training a network according to some embodiments.

[0039] Fig.14A is a flow chart of an example process for fine-tuning a pre-trained network using a custom kernel in accordance with some embodiments.

[0040] Fig. 14B is a flow chart of another example process for fine-tuning a pre-trained network using the folding and unfolding technique according to some embodiments.

[0041] Fig.15Depicted is an example implementation of a folding and unfolding technique for fine-tuning training in accordance with some embodiments.

[0042] Fig.16 A block diagram illustrating an example of components of an apparatus capable of implementing various aspects of the present disclosure is shown.

[0043] Like reference numbers and designations in the various drawings indicate like elements. DETAILED DESCRIPTION

[0044] Neural networks and other machine learning models are being used to generate predictions related to data. For example, for audio data, a trained machine learning model can be used to classify audio signals into speech or noise for speech recognition, emotion detection of speech in audio data, and so on. However, such a network may have difficulty in making predictions for streaming data in which data arrives sequentially in real time or near real time. For example, transformer networks have been used to perform functions such as speech recognition and text translation using multi-headed attention modules that generate attention vectors based on all data. As a more specific example, using a traditional transformer network, an attention vector can be generated for a portion of an input speech or text string, which indicates the context of the portion relative to previous and future portions of the speech or text. However, this is not possible for streaming data because in streaming data, future data (i.e., the remainder of a sentence, speech after a given word or phrase) is not easily accessible.

[0045] Disclosed herein are techniques (e.g., systems, methods, and media) for implementing a network utilizing a streaming attention mechanism. Specifically, using the techniques described herein, an attention vector can be generated based on a streaming data window, rather than generating an attention vector based on all data as in a traditional transformer network. A multi-head streaming attention block can be used to generate an attention vector. Similar to a traditional transformer network, a multi-head streaming attention block can be configured to generate an attention vector based on a query, key, and value transformation. The query transformation can generally indicate a set of vectors for which attention is to be calculated, while the value transformation can generally indicate a set of vectors for which attention is to be calculated. Therefore, by performing a dot product multiplication of the query and the value, a set of unnormalized weights can be generated that indicate the degree of attention to each query relative to the key. The set of unnormalized weights can then be normalized using, for example, a softmax function, and the normalized weights can be multiplied by the value to obtain an output. Unlike a conventional multi-head attention block of a conventional transformer network, a multi-head streaming attention block can be configured to maintain and update query, key, and value buffers frame by frame, such that new query, key, and value transformations are updated in the buffers and old values ​​are discarded. Unlike a conventional transformer network that utilizes all query data to generate an attention vector, as disclosed herein, a multi-head streaming attention block can use a window of query data to generate an attention vector.

[0046] In some embodiments, the attention vector may be generated using a frame reasoning method, wherein the query, key, and value buffers are updated based on the current frame of streaming data, and wherein a single query frame is used to generate the attention vector. An exemplary multi-head streaming attention block for implementing the frame reasoning method is shown in FIG. Figure 6 Shown and combined Figure 6 Additionally or alternatively, in some embodiments, the attention vector may be generated using a block reasoning method, where the query, key, and value buffers are updated based on the current frame of the streaming data and one or more look-ahead frames. The attention vector may be generated based on multiple query frames (e.g., the current query frame and one or more look-ahead frames). Fig. 9 An example multi-head streaming attention block (sometimes also referred to herein as a low-latency multi-head streaming attention block) for implementing a block reasoning method is shown and described. It should be noted that the frame reasoning method may reduce computational complexity relative to the block reasoning method, and the block reasoning method may have lower latency relative to the frame reasoning method. Therefore, a method can be selected based on the computational power of the computing device to be used to generate the reasoning, the type of streaming data to be used, etc.

[0047] It should be noted that although the technology disclosed herein is generally described for streaming audio data, other types of streaming data are also considered. In some embodiments, other types of streaming data, such as streaming text data, etc., can be used.

[0048] The network trained to make predictions associated with the streaming data may make any suitable predictions. For example, where the streaming data is streaming audio data, the predictions may include real-time or near real-time predictions of speech recognition, emotion detection, speaker detection or recognition, speech representation learning, etc. In some embodiments, generating predictions in real-time or near real-time may allow the model to be updated without utilizing stored user data to generate the predictions, or without storing data for a longer period of time than is necessary to generate the predictions. This may help protect user privacy by not storing, for example, audio conference or video conference data.

[0049] Figure 1 An example audio environment 100 is shown. Figure 1 An audio device 104 is included, which can be configured to capture an audio signal 101 via a plurality of microphones (e.g., microphones 106a, 106b, and 106c). An example of such an audio device is a device for audio conferencing or video conferencing, such as can be used in a conference room. The audio device 104 can include one or more processors 105. The one or more processors 105 can be configured to analyze the audio signal and, optionally, any other signal (e.g., a video signal, data from a motion sensor, etc.). The one or more processors 105 can be configured to generate speech analysis, such as performing speech recognition, emotion recognition, speaker detection, etc. The speech analysis can be generated in real time or near real time based on streaming audio data. Disclosed herein is a technique that can be executed by one or more processors 105 or one or more processors communicatively coupled to one or more processors 105 (e.g., by a processor of a server device or other coupled computing device) to perform or generate such speech analysis.

[0050] As described above, the present invention discloses a technique for implementing a streaming attention mechanism. The technique can be used in combination with various types of neural network architectures (e.g., transformer networks, conformer networks, etc.). Such a neural network can be trained, for example, by a training set. Once the training is completed, a network using a streaming attention mechanism (described in detail below) implemented by a multi-head streaming attention layer can be used to generate predictions for input streaming data during the inference phase. For example, the input streaming data can be streaming audio data (e.g., streaming voice signals, etc.). Predictions associated with streaming audio data may include real-time or near real-time speech recognition, speech emotion recognition, speaker detection or recognition, speech representation learning, etc. In some embodiments, the streaming data can be streaming text data, video data, financial data, accelerometer data, power usage data, or any other type of time series data.

[0051] It should be understood that training a neural network configured to implement a streaming attention mechanism can be performed by a different device than a device that performs inference using the trained neural network. For example, training can be performed by a first server device, and inference can be performed using the trained neural network by an end-user device (e.g., an audio or video conferencing device, a laptop, a desktop computer, etc.), a second server device, etc. Figure 4 , 5 , 6, 7, 8, 9, 10A, 10B, 11 and 12 show and describe more detailed techniques for implementing the streaming attention mechanism. Fig.13 , 14A , 14B and 15 show and describe techniques for training networks using streaming attention mechanisms.

[0052] Figure 2 A flow chart of an example process 200 for training a machine learning model and utilizing a trained machine learning model according to some embodiments is depicted. The machine learning model may be a model configured to implement a streaming attention mechanism and generate predictions about streaming data during an inference phase. It should be noted that in some embodiments, the blocks of process 200 may be implemented by multiple devices. For example, training a machine learning model may be implemented by a first device or a first group of devices, and reasoning using the machine learning model may be implemented by a second device or a second group of devices. As another example, in some embodiments, training the model may be performed by one or more server devices, and reasoning may be performed by one or more client devices or terminal computing devices (e.g., conferencing devices, laptops, desktop computers, etc.). In some embodiments, the blocks of process 200 may be performed in accordance with Figure 2 In some embodiments, two or more blocks of process 200 may be performed substantially in parallel. In some embodiments, one or more blocks of process 200 may be omitted.

[0053] At 202, process 200 can obtain a training set. The training set can include training samples, where each training sample can include an input to a machine learning model and a corresponding true value prediction. For example, in the case of training a machine learning model to perform speech recognition on an input speech signal, the training set can include training samples, each training sample consisting of a block of an input audio signal including speech and corresponding text indicating a word contained in the input audio signal.

[0054] At 204, process 200 may train a machine learning model using the training set. The machine learning model may be a transformer network, a conformer network, etc. The machine learning model may have an architecture including a multi-head streaming attention layer or block, as described below in conjunction with Figure 5 and Figure 6 It should be noted that Fig.13 , 14A , 14B and 15 describe techniques for pre-training a model and then fine-tuning the model to generate a trained model. In addition, it should be noted that the trained machine learning model can be characterized by a set of weights (e.g., weights associated with different nodes and / or layers of the network), where the set of weights is the weight value at the end of the training phase.

[0055] At 206, process 200 can make one or more predictions using the trained machine learning model in the inference phase. One or more predictions can be predictions related to a stream data block (e.g., a stream audio data block). For example, one or more predictions can include speech recognition related to a stream audio data block, sound emotion recognition related to a stream audio data block, speaker detection related to a stream audio data block, etc. It should be noted that block 206 can be performed by a device different from the device that executes block 204. For example, block 206 can be performed by a terminal client device (e.g., a mobile user device, a desktop computer, a conference device, etc.), while block 204 can be performed by a server device. As another example, block 206 can be performed by a cloud device or a server device different from the server device that performs model training at block 204. For example, a cloud device or a server device can receive a stream data block from a client device (e.g., a video or audio conferencing device), and can process the stream data on the cloud to generate one or more predictions. In some implementations, one or more predictions can be transmitted back to the client device.

[0056] Figure 3A and 3B FIG. 1 shows the inference phase applied to streaming data according to some embodiments. Figure 2 A and 2B generally depict streaming audio data, but the techniques illustrated and described below can be applied to other types of streaming data.

[0057] Go to Figure 3A , components of the inference block 300 are depicted according to some embodiments. As shown, an input audio signal may be provided to a feature extraction block 302. The input audio signal may be obtained from one or more microphones. The input audio signal may be streamed audio data such that the audio arrives in block order, wherein each block includes a fixed number of samples. For example, the inference block 300 may run at a sampling rate of 16,000 samples per second and may receive 10 milliseconds (e.g., 160 samples) of audio at a time. It should be noted that in some systems, it may be required that an audio block must be processed within a block time. For example, if the audio is received in 10 millisecond frames, each frame must be processed within 10 milliseconds in order to output a prediction at the same rate as the input frame. The feature extraction block may be configured to extract a feature vector associated with the input acoustic audio signal.

[0058] The extracted feature vectors may be provided to an inference model 304. The inference model 304 may include a trained machine learning model incorporating a streaming attention layer or block, as described below. The inference model 304 may generate one or more predictions associated with the input audio signal as output.

[0059] Figure 3B A schematic diagram of extracting feature vectors and providing the feature vectors to an inference model according to some embodiments is depicted. As shown, at 352, an input audio signal is provided to a feature extraction block. At 354, one or more feature vectors are extracted. The one or more feature vectors are provided to an inference model. At 356, the inference model generates one or more predictions associated with the input audio signal. It should be noted that, as Figure 3B As shown, the input audio signal is a sequential signal and may last for any suitable length of time, such as one second, ten seconds, one minute, ten minutes, thirty minutes, one hour, ten hours, etc. A prediction may be generated sequentially for each frame of the input audio signal. Figure 3B In the example, the sequential prediction flow is represented as L 1 , L 2 , L 3 etc., where L n represents the prediction associated with the nth frame.

[0060] The inference model used to generate predictions may include various blocks or layers. For example, the inference model may include a position encoding layer configured to embed position information associated with a block of streaming data. For example, the position encoding layer may indicate position information of a particular word or speech token within the data stream. As a more specific example, the position information may indicate an absolute position (e.g., relative to the beginning of the data stream) or a relative position (e.g., relative to a current point in time). As a more specific example, the position information may be combined with the extracted features to generate input data provided to a transformer layer. The transformer layer may include a streaming multi-head attention block, such as Figure 5 As shown and combined below Figure 5 The output of the transformer layer may then be provided to a linear layer and then to a softmax layer. The linear layer may be configured to perform a linear transformation on the output of the transformer layer. The output of the softmax layer may be a prediction associated with a block of streaming data. In other words, the softmax layer may be configured to assign probabilities to a set of candidate predictions, and a prediction may then be selected from the set of candidate predictions based on the probabilities assigned by the softmax layer.

[0061] Figure 44 is a diagram of an example implementation of an inference model according to some embodiments. As shown, the extracted features 402 and the position information generated by the position encoding layer 404 can be combined at 404 to generate transformer input data. The transformer input data can be provided to a repeated transformer 406 with a streaming attention block, which can be configured to generate and / or utilize a streaming attention vector. The output of the transformer 406 with a streaming attention block can be provided to a linear layer 408, which can be configured to perform a linear transformation on the output of the transformer with a streaming attention layer. The output of the linear layer can be provided to a softmax layer 410, which can be configured to determine the probability associated with a set of candidate predictions. For example, in the case where the inference model is configured to perform speaker detection on a set of N candidate speakers, the softmax layer can assign the probability of the streaming audio data block being associated with each of speakers 1, ..., N. Then, the prediction block 412 can be configured to select a prediction from the candidate prediction set based on the probability assigned by the softmax layer 410.

[0062] Figure 5 An example implementation of a transformer layer of an inference model according to some implementations is shown. As shown, an input can be provided to a layer normalization block 502. It should be noted that the layer normalization block 502 is optional and can be omitted. The output of the layer normalization block 502 (or, if the layer normalization block 502 is omitted, the input) can be provided to a streaming multi-head attention block 504. The streaming multi-head attention block 504 can be configured to generate an attention vector based on the streaming data block. Specifically, as described below in conjunction with Figure 6 and 9 In more detail, the streaming multi-head attention block 504 can generate an attention vector based on query, key, and value transformation values ​​obtained from corresponding query, key, and value buffers. Each buffer can be a circular buffer that is configured to update and / or replace values ​​based on sequential streaming data, thereby allowing attention vectors to be generated based on recent streaming data. Figure 6 (e.g., a frame inference method configured to implement streaming attention) and Fig. 9 (e.g., a block inference method configured to implement streaming attention) shows and describes an example implementation of a streaming multi-head attention block. The output of the streaming multi-head attention block 504 can be provided to a position feed-forward layer 506, which can be configured to generate a transformer layer output, which in turn can be provided to a linear layer, as described above in conjunction with Figure 4 As shown and described.

[0063] In some embodiments, the multi-head streaming attention block may include a query buffer, a key buffer, and a value buffer. Each of the query buffer, the key buffer, and the value buffer may be configured to store values ​​associated with the streaming data frame, so that the query buffer, the key buffer, and the value buffer are updated based on sequential frames. For example, for a given streaming data frame, a query transformation, a key transformation, and a value transformation may be identified based on input data for the streaming data frame. It should be noted that in some embodiments, any of the query transformation, the key transformation, and the value transformation may be a matrix. The query buffer, the key buffer, and the value buffer may be updated based on the identified query, key, and value transformations. For example, a transformation associated with the current frame may be added to the buffer, while an older transformation may be deleted from the buffer. In some implementations, each buffer may be a circular buffer. In some embodiments, one or more query frames may be retrieved from the updated query buffer. A dot product may be determined based on the retrieved one or more query frames and the frame in the key buffer to determine a set of weights. This set of weights may be used to determine a streaming attention vector.

[0064] Figure 6 Depicted is an example implementation of a multi-head streaming attention block in accordance with some embodiments. Figure 6 The multi-head streaming attention block shown can be used to implement a frame reasoning method for streaming attention. As shown, an input 601 is provided. The input 601 can correspond to a streaming data frame (e.g., a streaming audio data frame). The input 601 is provided to a query projection 602, a key projection 603, and a value projection 604. Each of the query projection 602, the key projection 603, and the value projection 604 can generate a linear projection of the input 601 to the query space, the key space, and the value space, respectively. The projections from the query projection 602, the key projection 603, and the value projection 604 can be provided to the multi-head attention segmentation 615. Traditionally, the dot product product between the query value and all key frames is determined. However, in order to perform streaming reasoning, the dot product product is determined using the delayed query and key. The delay of the query value is performed by a first-in, first-out (FIFO) delay management block 606, and the delay of the key value is performed by a historical delay management block 607. It should be noted that the example implementation of the FIFO delay management block 606 is Figure 7 As shown in and combined with the following Figure 7 , and an example implementation of the historical delay management block 607 is described in Figure 8 As shown in and combined with the following Figure 8Described. The dot product of the delayed query transformation and the delayed key transformation is performed by the dot product block 609. The output of the dot product block 609 is a set of weights. The delayed version of the value transformation is generated by the historical delay management block 608. The weighted sum block 610 uses the delayed value transformation and the weight set to generate a weighted sum. It should be noted that block 605 can be repeated for each head of the multi-head streaming attention block. The output of each head is spliced ​​by the splicing block 611. The delayed version of the input 601 is generated by the FIFO delay management block 612, and the output of the delayed input and the spliced ​​sum are provided to the add residual block 613. Output 614 corresponds to the output of the different heads of the multi-head attention layer with a delayed version of the residual.

[0065] The FIFO delay manager may be used to delay query transformations and inputs. The FIFO delay manager may be configured to store the current frame representation (e.g., Figure 6 The FIFO delay manager block 612 is the case of the current input frame, or as Figure 6 The FIFO delay management block 606 (the current query projection in the case of the FIFO delay management block 606) is attached to the front end of the stack and the last frame of the stack is discarded, thereby making the FIFO delay manager operate as a queue. In some embodiments, the queue can be configured as a circular buffer so that older frames are deleted and / or overwritten.

[0066] Figure 7 An example implementation of a FIFO delay management block 700 according to an embodiment is shown. In some embodiments, the FIFO delay management block 700 can be implemented as a FIFO delay management block 606 and / or 612. As shown, an input frame 701 can be attached to the front end of a delay line 702. In some embodiments, the delay line 702 can operate with a fixed amount of memory (e.g., implemented using a circular buffer and / or a queue) so that when a new frame (e.g., input frame 701) is added to the head of the delay line 702, the old frame (e.g., old frame 703) is discarded. The output frame 504 can be determined based on the query delay value. In some embodiments, the output frame 504 can be the same as the discarded frame 503, so that the amount of memory occupied by the delay line 502 is minimized.

[0067] The history delay management block can be configured to generate delayed versions of value and key transformations, such as Figure 6 As shown and combined with the above Figure 6 Unlike the FIFO delay management module, the historical delay management module can be configured to discard the oldest frame and then output the entire remaining stack instead of just a selected frame.

[0068] Figure 8FIG. 8 shows an example implementation of the historical delay management block 800 according to some embodiments. In some embodiments, the historical delay management block 800 may be implemented as the historical delay management block 607 and / or 608, such as Figure 6 Combined with the above Figure 6 As shown, input frame 801 can be attached to the front end of delay line 802. Delay line 802 can be implemented as a circular buffer and / or queue. In order to operate the delay line with a fixed amount of memory, when input frame 801 is added to the front end of delay line 802, the oldest frame 803 is discarded. It should be noted that the output of historical delay management block 800 includes all delayed frames 802 after appending input frames and discarding the oldest frame 803.

[0069] exist Figure 6 As shown in the above Figure 6 In the described multi-head attention layer, for a given frame, the current query block is dot-multiplied with all key blocks. However, for networks with multiple layers, this may result in longer latency, which may cause problems when generating predictions related to streaming data. For example, given a six-layer transformer network operating on two frames of audio (e.g., a look-ahead frame and a current frame), given a frame duration of 20 milliseconds, the total latency of the six layers may be 2*20 milliseconds*6 layers, or 240 milliseconds. This article discloses a low-latency implementation of a multi-head streaming attention block configured to generate a streaming attention vector using a block reasoning method. In addition to multiplying the query block of the current frame with all key blocks, multiple query blocks (e.g., the current query block and N look-ahead blocks, where N is 2, 3, etc.) can also be multiplied with all key blocks. The number of look-ahead blocks N may be the length of a FIFO delay management block (e.g., the length of a delay line). For each consecutive layer of the transformer, input can be provided from the previous layer, where all future blocks can be replaced with new versions of the look-ahead blocks generated from the previous layer. Replacing future blocks based on the output of previous layers may prevent latency accumulation. Given the example of an audio frame of 20 milliseconds duration and a six-layer transformer described above, a multi-headed streaming attention block considering two frames (one current frame and one look-ahead frame) may have a latency of 2*20 milliseconds, or 40 milliseconds, rather than the latency of 100 milliseconds. Figure 6 As shown in the above Figure 6 The 240 ms latency associated with the use of the described multi-head attention layer.

[0070] Fig. 9 Depicted is an example implementation of a low-latency multi-head streaming attention block that can be used to implement a block reasoning approach in accordance with some embodiments. Fig. 9 Many components of the low-latency multi-head attention layer shown are similar to Figure 6 The corresponding components of the multi-head attention layer shown in are similar. However, Figure 6The difference shown is that the FIFO delay management block and the history delay management block are implemented as block delay blocks. For example, FIFO delay management blocks 606 and 612 are replaced by block FIFO delay management blocks 906 and 912, and history delay management blocks 607 and 608 are replaced by block history delay management blocks 907 and 908.

[0071] Block FIFO delay management blocks (e.g., block FIFO delay management blocks 906 and 912) may be used to identify delayed query and input blocks. A first version of the block FIFO delay management block may be used in the first layer of the network, where in the first version, incoming input frames are appended to the delay line and the oldest frame is discarded. The output of the first frame may be multiple frames, for example, a current frame and N look-ahead frames, where N is 1, 2, 3, 4, etc. frames. A second version of the block FIFO delay management block may be used in subsequent layers of the network, where the input to the delay line is the same as the output. Propagation of the input to the output may prevent delays from accumulating in successive layers of the transformer network.

[0072] Fig. 10A An example implementation of a block FIFO delay management block that may be used in conjunction with a network first layer according to some embodiments is shown. As shown, a current frame 1001 is appended to a delay line 1002. The delay line 1002 may be implemented as a circular buffer, a queue, etc. The oldest frame 1003 is discarded from the end of the delay line 1002. The output 1004 is all frames of the delay line after the oldest frame 1003 is discarded. Fig. 10B An example implementation of a block FIFO delay management block that may be used in conjunction with subsequent layers of a network according to some embodiments is shown. As shown, output 1054 is identical to input 1051 .

[0073] It should be understood that although Fig. 10A and 10B The delay line shown in includes one current frame and two look-ahead frames, but this is just an example. In some embodiments, the delay line can be configured to operate with a current frame and N look-ahead frames, where N is 1, 2, 4, 5, etc. In any case, for a low-latency implementation of a multi-head streaming attention block, the block FIFO delay management block can be configured to output a number of frames corresponding to the current frame plus N look-ahead frames.

[0074] The block history delay management block can be used to maintain a buffer for key and value transformations in a low-latency multi-head attention layer. The block history delay management block can be configured to append and / or replace a set of frames in a delay line (e.g., a current frame and N look-ahead frames) and discard the oldest frame in the delay line. For example, the current frame can be appended to the delay line, and N look-ahead frames can replace N frames in the delay line. To operate with fixed memory, the oldest frame can be discarded from the delay line.

[0075] Fig.11An example implementation of a block history delay management block that can be used for some implementations of a low-latency multi-head streaming attention block according to some embodiments is shown. As shown, a set of frames 1101 may include a current frame and N forward frames. Fig.11 In the example shown, N is 2 and the set of frames 1101 includes a total of 3 frames. The current frame is appended to the delay line 1102 and the two look-ahead frames replace the two frames in the delay line 1102. The oldest frame 1103 is discarded. After updating the delay line (e.g., a circular buffer or queue), the key and / or value transformations indicated in the updated delay line can be used to generate a set of weights and / or weighted sums, such as Fig. 9 As shown and combined above Fig. 9 described.

[0076] In some embodiments, attention vectors can be generated for streaming data on a frame-by-frame basis. For example, at inference time, attention vectors can be generated based on a window of input frames rather than all input frames (e.g., as traditionally used to generate attention vectors for non-streaming data). In some embodiments, attention vectors can be generated by a multi-headed streaming attention block that manipulates time delays, generally referred to herein as a "frame inference method." For example, Figure 6 As shown in , this implementation can use the FIFO delay management block to delay the query transformation frame and the history delay management block to delay the key and value transformation frames. Then, multi-head streaming attention blocks can be stacked for each layer to generate an attention vector. For another example, Fig. 9 As shown in , the attention vector can be generated by a multi-head streaming attention block, which reduces latency by preventing latency across multiple layers of stacking, which is generally referred to as the "block reasoning method" in this article. Fig. 9 As shown and combined with the above Fig. 9 As described above, using the block inference method, a look-ahead frame can be inferred for each layer to prevent delay accumulation.

[0077] Whether using a frame reasoning method or a block reasoning method, an attention vector can be generated frame by frame and by updating a buffer associated with a query, key, and value transformation (e.g., a matrix). For example, for each frame of streaming data, a query, key, and value transformation can be identified based on the input data associated with the frame. The query, key, and value buffers can be updated based on the query, key, and value transformations so that the query, key, and value buffers can be configured to store values ​​associated with previous streaming data frames. After updating the buffer, one or more query frames can be retrieved from the updated query buffer to process the input data. For example, in a frame reasoning method, one query frame can be used to process the input data, while in a block reasoning method, multiple query frames (e.g., corresponding to the current query frame and one or more look-ahead frames) can be used. The dot product of one or more query frames and frames in the key buffer can be determined to determine a set of weights. The weighted sum can then be determined using the set of weights and frames in the value buffer. The weighted sum can then be used to determine a streaming attention vector, for example, by applying a softmax function to the weighted sum. The streaming attention vector can then be used to generate predictions associated with the streaming data.

[0078] Fig.12 1 is a flow chart of an example process 1200 for generating a streaming attention vector. As shown, the blocks of process 1200 may be performed by one or more control systems and / or processors, such as a server device, a video or audio conferencing device, a laptop device, etc. In some embodiments, the blocks of process 1200 may be performed in accordance with Fig.12 In some embodiments, two or more blocks of process 1200 may be executed in a different order than the order shown. Fig.12 In some embodiments, one or more blocks of process 1200 may be omitted.

[0079] Process 1200 may begin at 1202 by obtaining input data representing a stream data frame. The stream data frame may be obtained by a laptop device, a desktop computing device, an audio or video conferencing device, etc. The stream data may include stream audio data acquired by one or more microphones. The stream data may include speech or other audio data predicted by a network (e.g., a transformer network). It should be noted that the input data representing the stream data frame may include one or more extracted features associated with the frame, a position code associated with the frame, etc.

[0080] At 1204, process 1200 can identify queries, keys, and value transformations based on the input data of the streaming data frame. In some embodiments, the queries, keys, and value transformations can be in the format of query, key, and value matrices. In some implementations, techniques implemented by a non-streaming network (e.g., a non-streaming transformer network) can be used to identify queries, keys, and value transformations.

[0081] At 1206, process 1200 can update the query buffer, key buffer, and value buffer based on the query, key, and value transformations. The query, key, and value buffers can each be configured to store parameters associated with a previous frame of stream data. For example, in a frame inference approach, each buffer can be updated based on parameters associated with the current frame, such as Figure 6 , 7 and 8 and above combined Figure 6 , 7 As another example, in the block inference method, each buffer can be updated based on parameters associated with the current frame and one or more forward frames, as described above in conjunction with Fig. 9 , 10 and 11.

[0082] At 1208, process 1200 may retrieve one or more query frames from the updated query buffer for use in processing the input data. For example, using a frame inference method, a query frame may be retrieved, such as Figure 7 As shown and combined with the above Figure 7 As another example, using the block reasoning method, multiple query frames can be retrieved, such as Fig. 10A and 10B As shown and combined with the above Fig. 10A and 10B described.

[0083] At 1210, a dot product of the retrieved one or more query frames and the key frames in the key buffer may be determined to generate a set of weights. In a frame reasoning approach, the dot product may include a dot product of a single retrieved query frame and the key frames in the key buffer. Using a block reasoning approach, the dot product may include a dot product of multiple retrieved query frames and the key frames in the key buffer.

[0084] At 1212, a weighted sum between the set of weights determined at block 1210 and the frame in the value buffer may be determined. The weighted sum may correspond to an unnormalized attention vector.

[0085] Note that blocks 1208-1212 may be repeated for each layer of the network. For example, for a six-layer transformer network, blocks 1208-1212 may be repeated for each of the six layers. Each of blocks 1208-1212 may be performed by a multi-head streaming attention block (e.g., Figure 6 And / or 9 shown in combination with the above Figure 6 and / or 9), wherein each layer is associated with a different head.

[0086] At 1214, the weighted sum may be used to generate a streaming attention vector. For example, the unnormalized streaming attention vector generated at block 1212 may be normalized by applying the unnormalized streaming attention vector to a linear layer and / or a softmax layer, such as Figure 4 As shown and combined Figure 4 The output of the softmax layer may be a normalized streaming attention vector, which may be used to generate one or more predictions associated with the streaming data frame.

[0087] In some embodiments, a multi-stage training process can be used to train a network that utilizes a streaming attention mechanism (e.g., by implementing a multi-head streaming attention block, such as Figure 6 , 9 and 12 and above combined Figure 6 , 9 and 12). For example, a network can be trained by first training a version of the network that utilizes a non-streaming attention mechanism. As a more specific example, such a network can utilize a non-streaming multi-head attention block that does not use query, key, and value buffers. In such a non-streaming network, attention vectors can be generated using all data frames instead of frame windows. The weights of such a network can be updated based on backpropagation and / or any other suitable training technique. Fig.13 Here is an example flowchart for pre-training a network with streaming attention.

[0088] After pre-training a version of the network without streaming attention, the model can be fine-tuned using the fine-tuning training phase. Different techniques can be used to fine-tune the network. For example, Fig.14A As shown and combined below Fig.14A As described, fine-tuning can be performed using back-propagation to update the weights associated with the streaming attention kernel of the second version of the network including the query, key, and value buffers. As another example, Fig. 14B As shown and combined below Fig. 14B As described, fine-tuning can be performed by providing a series of time-shifted segments to a version of the network without streaming attention and updating the weights based on an aggregated set of predictions associated with the time-shifted segments.

[0089] Fig.13 1 is a flow chart of an example process 1300 for pre-training a streaming attention network using a version of the network that does not include one or more streaming attention blocks. The blocks of process 1300 may be implemented by one or more control systems and / or processors of a device such as a server device or other computing device suitable for training a model. In some embodiments, the blocks of process 1300 may be implemented in accordance with Fig.13In some embodiments, two or more blocks of process 1300 may be performed substantially in parallel. In some embodiments, one or more blocks of process 1300 may be omitted.

[0090] At 1302, process 1300 can obtain training data. The training data may be of the same type as data that a trained network configured to generate predictions for streaming data takes as input. For example, where the trained network is configured to operate on streaming audio data, the training data may correspond to frames of audio data. As a more specific example, where the trained network is configured to generate predictions related to speech in streaming audio data, the training data may include frames of speech audio data.

[0091] At 1304, process 1300 can generate a prediction output using a machine learning model that utilizes a non-streaming attention mechanism. For example, the machine learning model can be a traditional transformer network that does not include a multi-head streaming attention block. As a more specific example, the machine learning model can generate an attention vector based on all query frames instead of generating an attention vector based on a query frame window.

[0092] At 1306, process 1300 can determine a loss based on the training data and the predicted output. For example, the training data can include corresponding true output labels, and the loss can be determined based on the difference between the predicted output and the true output labels.

[0093] At 1308, process 1300 can use back propagation to update the weights associated with the model. Blocks 1302-1308 can be repeated until a stopping criterion for network pre-training is reached. For example, the stopping criterion can include a loss less than a predetermined threshold, a change in weight after each iteration less than a predetermined threshold, etc.

[0094] Fig.14A is a flow chart of an example process 1400 for performing a pre-training process of a model configured to operate on streaming data to generate streaming attention vectors (e.g., Fig.13 As shown and combined above Fig.13 After the pre-training process described above is completed, the model is fine-tuned. In some embodiments, the blocks of process 1400 can be performed by one or more control systems or processors, such as a server device or other computing device suitable for training the model. In some implementations, the computing device performing process 1400 can be the same computing device as the computing device performing process 1300, or a different computing device. In some embodiments, the blocks of process 1400 can be performed in accordance with Fig.14AIn some embodiments, two or more blocks in process 1400 may be performed substantially in parallel. In some embodiments, one or more blocks in process 1400 may be omitted.

[0095] At 1402, process 1400 can obtain training data. The training data may be of the same type as data that a trained network configured to generate predictions for streaming data takes as input. For example, where the trained network is configured to operate on streaming audio data, the training data may correspond to chunks of audio data. As a more specific example, where the trained network is configured to generate predictions related to speech in streaming audio data, the training data may include chunks of speech audio data.

[0096] At 1404, process 1400 can generate a prediction output using a machine learning model that utilizes a streaming attention mechanism. For example, the machine learning model can have Figure 6 and Fig. 9 As shown and combined above Figure 6 and Fig. 9 The described multi-head streaming attention block. The machine learning model can be configured to generate a prediction output by generating a streaming attention vector using a window of frames instead of all previous frames.

[0097] At 1406, process 1400 can determine a loss based on the training data and the predicted output. For example, the training data can include corresponding true output labels, and the loss can be determined based on the difference between the predicted output and the true output labels.

[0098] At 1408, the process 1400 can use back propagation using derivatives derived from the streaming attention kernel to update the weights associated with the model. In some embodiments, the derivatives and the look-ahead portion for training can be determined based on the derived equations. The look-ahead portion and the derivatives can be implemented using a multi-threaded framework (e.g., using a compute unified device architecture (CUDA) framework, or other similar multi-threaded frameworks). It should be noted that the implementation can be called directly from a toolkit (e.g., PyTorch).

[0099] In some embodiments, fine-tuning of the pre-trained model can be achieved by providing a time-shifted input segment to the model and aggregating the output of the model in response to the time-shifted input segment. It should be noted that the model may not include a component configured to adapt to the stream input, such as a multi-head stream attention block. In addition, it should be noted that the techniques described herein for fine-tuning the pre-trained model using the time-shifted input segment can be applied to fine-tune any suitable model architecture (e.g., a transformer network, a consistent network, etc.) so that the model is suitable for using stream data as input. In some embodiments, fine-tuning can be performed by providing a series of time-shifted segments to the model for each input training sample to generate a corresponding series of predicted outputs. A series of predicted outputs can then be aggregated. Losses can be determined based on a series of predicted outputs after aggregation. The weights of the model (e.g., excluding the stream attention component) can then be updated based on the loss. It should be noted that the input data may include a kernel with several frames, wherein the several frames include several review frames, a current frame, and several forward-looking frames. The time shift of the segment may depend on a jump size parameter. In some embodiments, the jump size may correspond to a frame or block size. By temporally shifting the segments and providing them as separate inputs to the model, information mixing between frames can be avoided. It should be noted that in the case of block inference methods, multiple outputs corresponding to different delays from one layer to another can be passed layer by layer, and the last layer can generate a single output, from which the loss can be determined. The last layer may select a frame corresponding to the current time. In the case of frame inference methods, each layer may output a single frame so that the last layer also outputs a single frame. It should be noted that the outputs of different layers can be aggregated (e.g., using a weighted sum) before performing other operations (e.g., downstream tasks with different layers) or before using different loss functions.

[0100] Fig. 14B is a flow chart of an example process 1450 for fine-tuning a model that performs streaming attention inference by temporally shifting input segments, in accordance with some embodiments. In some embodiments, the blocks of process 1450 may be performed by one or more control systems or processors, such as a server device or other computing device suitable for training a model. In some implementations, the computing device that performs process 1450 may be the same computing device that performs process 1300, or a different computing device. In some embodiments, the blocks of process 1450 may be performed in accordance with Fig. 14B In some embodiments, two or more blocks of process 1450 may be performed substantially in parallel. In some embodiments, one or more blocks of process 1450 may be omitted.

[0101] Process 1450 may begin at 1452 with obtaining training data. The training data may be of the same type as data that a trained network configured to generate predictions for streaming data takes as input. For example, where the trained network is configured to operate on streaming audio data, the training data may correspond to chunks of audio data. As a more specific example, where the trained network is configured to generate predictions related to speech in streaming audio data, the training data may include chunks of speech audio data.

[0102] At 1454, process 1450 can generate a series of segments for data vectors included in training data including a look-back block (e.g., a previous data frame), a current block (e.g., a current frame of audio data), and a look-ahead block (e.g., a future frame), the series of segments including blocks shifted in time for a duration corresponding to the current block. It should be noted that in instances where the data corresponds to audio data, the duration corresponding to the current block can be a frame duration (e.g., 10 milliseconds, 12 milliseconds, 20 milliseconds, etc.). The data vector is generally referred to herein as a kernel, and the duration of the shift is generally referred to herein as a jump size.

[0103] Go to Fig.15 , where an input data block 1502 is shown. From the input data block 1502, data vectors corresponding to one or more review blocks, a current block, and one or more look-ahead blocks can be identified (e.g., as part of the input data block 1502). Panel 1504 shows a series of segments that can be generated based on the data vectors, where each segment is time-shifted by a duration corresponding to the duration of the current block, such as Fig.15 shown.

[0104] return Fig. 14B At 1456, process 1450 may generate a set of predicted outputs by providing each segment to the model. Note that, as described above, the model may not include any components that process streaming data, such as a multi-head streaming attention block. For example, the model may be a conventional transformer network. One output may be generated for each input segment.

[0105] Go to Fig.15 , panel 1506 shows a series of fragments being arranged in batches before being provided to the model.

[0106] return Fig. 14B At 1458, process 1450 can aggregate the series of prediction outputs. For example, aggregating the series of prediction outputs can include applying different loss functions.

[0107] Go to Fig.15 , panel 1508 illustrates a set of aggregated prediction outputs, according to some embodiments, which are aggregated based on the time shift of the corresponding time shift segments of panel 1504.

[0108] return Fig. 14B At 1460, process 1400 can determine a loss based on the aggregated series of prediction outputs. For example, the loss can be determined based on the difference between the aggregated series of prediction outputs and the true value prediction associated with the training data.

[0109] At 1462 , process 1450 may update weights associated with the model using back-propagated derivatives derived from the model that does not include a streaming attention component and based on the loss determined at 1460 .

[0110] After fine-tuning the model, individual components of the model may be replaced by streaming attention versions. Fig. 14B The multi-head attention block of a conventional Transformer network fine-tuned using the techniques described in can be replaced with a multi-head streaming attention block, e.g. Figure 6 and / or 9 and in combination with the above Figure 6 and / or as described in 9.

[0111] Fig.16 is a block diagram showing an example of components of an apparatus capable of implementing various aspects of the present disclosure. Figure 1 Sample, Fig.16 The types and quantities of the elements shown in are provided by way of example only. Other implementations may include more, less and / or different types and quantities of elements. According to some examples, device 1600 may be configured to perform at least some of the methods disclosed herein. In some implementations, device 1600 may be or may include one or more components of a television, an audio system, a mobile device (e.g., a cellular phone), a laptop, a tablet device, a smart speaker, or other types of devices.

[0112] According to some alternative implementations, the apparatus 1600 may be a server, or may include a server. In some such examples, the apparatus 1600 may be an encoder, or may include an encoder. Thus, in some cases, the apparatus 1600 may be a device configured for use within an audio environment (e.g., a home audio environment), while in other cases, the apparatus 1600 may be a device configured for use in the "cloud" (e.g., a server).

[0113] In this example, the device 1600 includes an interface system 1605 and a control system 1610. In some implementations, the interface system 1605 can be configured to communicate with one or more other devices in the audio environment. In some examples, the audio environment can be a home audio environment. In other examples, the audio environment can be another type of environment, such as an office environment, a car environment, a train environment, a street or sidewalk environment, a park environment, etc. In some implementations, the interface system 1605 can be configured to exchange control information and related data with the audio devices of the audio environment. In some examples, the control information and related data can relate to one or more software applications being executed by the device 1600.

[0114] In some implementations, the interface system 1605 may be configured to receive or provide a content stream. The content stream may include audio data. The audio data may include, but is not limited to, an audio signal. In some cases, the audio data may include spatial data, such as channel data and / or spatial metadata. In some examples, the content stream may include video data and audio data corresponding to the video data.

[0115] The interface system 1605 may include one or more network interfaces and / or one or more external device interfaces (e.g., one or more universal serial bus (USB) interfaces). According to some embodiments, the interface system 1605 may include one or more wireless interfaces. The interface system 1605 may include one or more devices for implementing a user interface, such as one or more microphones, one or more speakers, a display system, a touch sensor system, and / or a gesture sensor system. In some examples, the interface system 1605 may include a control system 1610 and a storage system (e.g., Fig.16 The interface system 1605 may include one or more interfaces between the control system 1610 and the optional storage system 1615 shown. However, in some cases, the control system 1610 may include a storage system. In some implementations, the interface system 1605 may be configured to receive input from one or more microphones in the environment.

[0116] The control system 1610 may include, for example, a general purpose single-chip or multi-chip processor, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic device, discrete gate or transistor logic and / or discrete hardware components.

[0117] In some implementations, the control system 1610 may reside in multiple devices. For example, in some implementations, a portion of the control system 1610 may reside in a device within one of the environments described herein, and another portion of the control system 1610 may reside in a device outside the environment, such as a server, a mobile device (e.g., a smartphone or tablet), etc. In other examples, a portion of the control system 1610 may reside in a device within an environment, and another portion of the control system 1610 may reside in one or more other devices of the environment. For example, a portion of the control system 1610 may reside in a device that implements a cloud-based service, such as a server, and another portion of the control system 1610 may reside in another device that implements a cloud-based service, such as another server, a storage device, etc. In some examples, the interface system 1605 may also reside in multiple devices. In some implementations, a portion of the control system may be located in or on an earbud.

[0118] In some implementations, the control system 1610 can be configured to at least partially perform the methods disclosed herein. According to some examples, the control system 1610 can be configured to implement methods for updating query, key, and value buffers, generating streaming attention vectors based on buffered query, key, and value parameters, and the like.

[0119] Some or all of the methods described herein may be performed by one or more devices according to instructions (e.g., software) stored on one or more non-transitory media. Such non-transitory media may include memory devices such as those described herein, including but not limited to random access memory (RAM) devices, read-only memory (ROM) devices, etc. One or more non-transitory media may reside, for example, on a computer. Fig.16 1610 and / or the control system 1610. Thus, various innovative aspects of the subject matter described in this disclosure can be implemented in one or more non-transitory media having software stored thereon. For example, the software can extract objects from a multi-channel audio signal, generate a spatial enhancement mask, apply a spatial enhancement mask, generate an output binaural audio signal, etc. The software can be, for example, executed by a control system (e.g. Fig.16 Executed by one or more components in the control system 1610).

[0120] In some examples, apparatus 1600 may include Fig.16Optional microphone system 1620 shown. Optional microphone system 1620 may include one or more microphones. In some implementations, one or more microphones may be part of or associated with another device, such as a speaker of a speaker system, a smart audio device, etc. In some examples, device 1600 may not include microphone system 1620. However, in some such embodiments, device 1600 may still be configured to receive microphone data from one or more microphones in the audio environment via interface system 1610. In some such implementations, a cloud-based implementation of device 1600 may be configured to receive microphone data or a noise metric corresponding at least in part to the microphone data from one or more microphones in the audio environment via interface system 1610.

[0121] According to some embodiments, the apparatus 1600 may include Fig.16 Optional speaker system 1625 is shown. Optional speaker system 1625 may include one or more speakers, which may also be referred to herein as "speakers," or more generally as "audio reproduction transducers." In some examples (e.g., cloud-based implementations), device 1600 may not include speaker system 1625. In some implementations, device 1600 may include headphones. The headphones may be connected or coupled to device 1600 via a headphone jack or a wireless connection (e.g., Bluetooth).

[0122] Some aspects of the present disclosure include systems or devices configured (e.g., programmed) to perform one or more examples of the disclosed methods, and tangible computer-readable media (e.g., optical disks) storing code for implementing one or more examples of the disclosed methods or steps thereof. For example, some disclosed systems may be or include a programmable general-purpose processor, a digital signal processor, or a microprocessor that is programmed and / or otherwise configured with software or firmware to perform any of a variety of operations on data, including embodiments of the disclosed methods or steps thereof. Such a general-purpose processor may be or include a computer system that includes an input device, a memory, and a processing subsystem that is programmed (and / or otherwise configured) to perform one or more examples of the disclosed methods (or steps thereof) in response to data asserted thereto.

[0123] Some embodiments may be implemented as a configurable (e.g., programmable) digital signal processor (DSP) that is configured (e.g., programmed or otherwise configured) to perform the desired processing on the audio signal, including performing one or more examples of the disclosed method. Alternatively, embodiments of the disclosed system (or its elements) may be implemented as a general-purpose processor (e.g., a personal computer (PC) or other computer system or microprocessor, which may include an input device and a memory), which is programmed and / or otherwise configured with software or firmware to perform any of a variety of operations, including one or more examples of the disclosed method. Alternatively, elements of some embodiments of the system of the present invention are implemented as a general-purpose processor or DSP configured (e.g., programmed) to perform one or more examples of the disclosed method, and the system also includes other elements (e.g., one or more speakers and / or one or more microphones). The general-purpose processor configured to perform one or more examples of the disclosed method may be coupled to an input device (e.g., a mouse and / or keyboard), a memory, and a display device.

[0124] Another aspect of the disclosure is a computer-readable medium (eg, a disk or other tangible storage medium) storing code (eg, an encoder executable to carry out the examples) for executing one or more examples of the disclosed method or steps thereof.

[0125] Although specific embodiments of the present disclosure and applications of the present disclosure have been described herein, it will be apparent to those skilled in the art that many modifications may be made to the embodiments and applications described herein without departing from the scope of the disclosure as described and claimed herein. It should be understood that although certain forms of the present disclosure have been shown and described, the present disclosure is not limited to the specific embodiments described and shown or the specific methods described.

Claims

1. A method for processing stream data, the method comprising: Get input data representing a stream data frame; Identify query transformations, key transformations, and value transformations based on input data of the streaming data frame; updating the query buffer, the key buffer, and the value buffer based on the identified query transformation, the key transformation, and the value transformation, such that the query buffer, the key buffer, and the value buffer are each configured to store parameters associated with a previous stream data frame and the stream data frame; retrieving one or more query frames from the updated query buffer for use in processing the input data; determining a dot product of the retrieved one or more query frames and a frame in the key buffer to determine a set of weights; determining a weighted sum between the set of weights and the frame in the value buffer; The weighted sum is used to generate a streaming attention vector, which the network can use to generate predictions associated with the streaming data.

2. The method according to claim 1, wherein the network is a converter network.

3. The method according to any one of claims 1 or 2, wherein the streaming data is streaming audio data.

4. The method of any one of claims 1 to 3, wherein the prediction associated with the stream data comprises a prediction of a speech emotion associated with the stream data.

5. A method according to any one of claims 1 to 4, wherein the prediction associated with the streaming data includes identification of one or more features that can be used to provide to one or more downstream machine learning models.

6. The method of claim 5, wherein the one or more characteristics include identification of one or more speakers associated with the streaming data.

7. The method of any one of claims 1 to 6, wherein the prediction associated with the streaming data comprises a classification of one or more words or phonemes of the streaming data.

8. The method of any one of claims 1 to 7, wherein at least one of the query buffer, the key buffer or the value buffer is a circular buffer.

9. The method according to any one of claims 1 to 8, wherein updating the query buffer comprises: appending the current query frame based on the query transformation to the query buffer; as well as The oldest query frame in the query buffer is discarded.

10. The method of claim 9, wherein the retrieved one or more query frames correspond to the oldest query frame that was discarded.

11. The method of any one of claims 1 to 8, wherein the network comprises a plurality of layers, and wherein updating the query buffer comprises, for a first layer of the plurality of layers: appending the current query frame based on the query transformation to the query buffer; replacing a plurality of query frames in the query buffer with a plurality of look-ahead query frames corresponding to future times; and The oldest query frame in the query buffer is discarded.

12. The method according to claim 11, wherein: The retrieved one or more query frames used by a first layer of the plurality of layers to process an input block include a current query frame and the plurality of look-ahead query frames.

13. The method according to any one of claims 11 or 12, wherein the plurality of look-ahead query frames comprises two look-ahead frames.

14. The method according to any one of claims 11 to 13, wherein: The retrieved one or more query frames used by each layer of the plurality of layers except the first layer to process the input block are passed to a given layer by a previous layer.

15. The method according to any one of claims 1 to 14, wherein updating the key buffer and updating the value buffer comprises: Append the current key frame based on the key transformation to the key buffer and discard the oldest key frame from the key buffer; The current value frame based on the value transform is appended to the value buffer; and the oldest value frame is discarded from the value buffer.

16. The method of any one of claims 1 to 14, wherein updating the key buffer and updating the value buffer comprises: Append the current key frame to the key buffer; Based on the key transformation, replacing a plurality of key frames in the key buffer with a plurality of look-ahead key frames; Discard the oldest key frame from the key buffer; Append the current value frame to the value buffer; replacing a plurality of value frames in a value buffer with a plurality of look-ahead value frames based on the value transformation; as well as Discard the oldest value frame from the value buffer.

17. A method according to any one of claims 1 to 16, wherein the network is trained by: performing initial training using a first version of the network that does not utilize a query buffer, a key buffer, and a value buffer to store a subset of query frames, key frames, and value frames; and Performing subsequent training that modifies the weights associated with the network.

18. The method of claim 17, wherein subsequent training is performed using a second version of the network comprising a query buffer, a key buffer, and a value buffer, and wherein performing subsequent training comprises performing back propagation using derivatives derived from the second version of the network.

19. The method of claim 17, wherein subsequent training is performed using the first version of the network, and wherein performing subsequent training comprises: providing a first version of the network with data for a given training data block, a series of time-shifted segments to generate a corresponding series of predicted outputs; Aggregate the series of forecast outputs; determining a loss based on the aggregated series of predicted outputs; and The weights associated with the first version of the network are updated based on the loss.

20. A system comprising: one or more processors; as well as A non-transitory computer readable medium storing instructions that, when executed by the one or more processors, cause the one or more processors to perform the operations of claims 1 to 19.

21. A non-transitory computer readable medium storing instructions that, when executed by one or more processors, cause the one or more processors to perform the operations of claims 1 to 19.

Images