Systems and methods for training a dual-mode machine learning speech recognition model

Abstract

Systems and methods of the present disclosure relate to a computing system comprising one or more processors and a machine learning multi-modal speech recognition model configured to operate in a streaming recognition mode or a contextual recognition mode. The computing system can operate comprising obtaining speech data and ground truth labels; and processing the speech data using the contextual recognition mode to obtain contextual prediction data. The operations can comprise evaluating a difference between the contextual prediction data and the ground truth labels; and processing the speech data using the streaming recognition mode to obtain streaming prediction data. The operations can comprise evaluating a difference between the streaming prediction data and the ground truth labels and the contextual and streaming prediction data. The operations can comprise adjusting parameters of the speech recognition model.

Description

[0001] Related applications

[0002] This application claims priority and benefit to U.S. Provisional Patent Application No. 63 / 086,937, filed October 2, 2020, which is incorporated herein by reference in its entirety. Technical Field

[0003] This disclosure generally relates to speech recognition. More specifically, this disclosure relates to training and utilizing a dual-mode machine learning speech recognition model. Background Technology

[0004] Speech recognition technology has become increasingly important for bridging the communication gap between humans and machines. Low-latency streaming speech recognition (e.g., online speech recognition, iterative speech recognition) is used in several scenarios where it is crucial to recognize each word as quickly as possible while it is spoken (e.g., virtual assistant tasks). While low-latency streaming recognition is often preferred for many speech recognition scenarios, it is typically subject to prediction errors (e.g., high word error rates). For scenarios where accuracy is critical (e.g., offline video captioning on video sharing platforms), full-context speech recognition is often preferred. In full-context speech recognition, the entire speech segment is provided as input, allowing earlier parts of the speech to be processed based on later context. Summary of the Invention

[0005] Aspects and advantages of embodiments of this disclosure will be set forth in part in the description which follows, or may be learned from the description or by practice of the embodiments.

[0006] One example aspect of this disclosure relates to a computer-implemented method for utilizing a multimodal speech recognition model. The method may include: obtaining a machine learning multimodal speech recognition model by a computing system including one or more computing devices, the machine learning multimodal speech recognition model being configured to operate in either a streaming recognition mode or a context recognition mode, wherein the streaming recognition mode is configured to iteratively encode each of one or more portions of speech data, and wherein the context recognition mode is configured to simultaneously encode each of one or more portions of speech data. The method may include: obtaining speech data corresponding to a speech recognition task by the computing system. The method may include: determining a selected mode from the streaming recognition mode and the context recognition mode of the machine learning multimodal speech recognition model by the computing system based on the speech recognition task. The method may include: processing additional speech data by the computing system using the selected mode of the machine learning multimodal speech recognition model to obtain a speech recognition output.

[0007] Another example aspect of this disclosure relates to a computational system for training a machine learning model for multimodal speech recognition. For example, a computer system can be used to train a machine learning model for multimodal speech recognition, as described in the foregoing aspects. The computational system may include one or more processors. The computational system may include a machine learning multimodal speech recognition model configured to operate in either a streaming recognition mode or a context recognition mode. The streaming recognition mode may be configured to iteratively encode each of one or more portions of speech data to obtain streaming prediction data (e.g., causally). The context recognition mode may be configured to simultaneously encode each of one or more portions of speech data to obtain context prediction data (e.g., non-causally). The computational system may include one or more tangible, non-transitory computer-readable media storing computer-readable instructions that, when executed by one or more processors, cause the one or more processors to operate. This operation may include: obtaining speech data and associated ground truth labels. This operation may include: processing the speech data using the context recognition mode of the machine learning multimodal speech recognition model to obtain context prediction data. This operation may include: evaluating a context loss function that assesses the difference between the context prediction data and the ground truth labels. This operation may include: processing speech data using a streaming recognition mode of a machine learning multimodal speech recognition model to obtain streaming prediction data. This operation may include: evaluating a streaming loss function that assesses the difference between the streaming prediction data and the ground truth labels, and the difference between the contextual prediction data and the streaming prediction data. This operation may include: adjusting one or more parameters of the machine learning multimodal speech recognition model, at least in part, based on the contextual loss function and the streaming loss function.

[0008] Another example aspect of this disclosure relates to one or more tangible, non-transitory computer-readable media storing computer-readable instructions that, when executed by one or more processors, cause the one or more processors to operate. This operation may include: obtaining a machine learning multimodal speech recognition model configured to operate in either a streaming recognition mode or a context recognition mode, wherein the streaming recognition mode is configured to iteratively encode each of one or more portions of speech data, and wherein the context recognition mode is configured to simultaneously encode each of one or more portions of speech data. This operation may include: obtaining speech data corresponding to a speech recognition task. This operation may include: determining a selected mode from the streaming recognition mode and the context recognition mode of the machine learning multimodal speech recognition model based on the speech recognition task. This operation may include: processing additional speech data using the selected mode of the machine learning multimodal speech recognition model to obtain a speech recognition output.

[0009] Other aspects of this disclosure relate to various systems, devices, non-transitory computer-readable media, user interfaces, and electronic devices.

[0010] These and other features, aspects, and advantages of the various embodiments of this disclosure will be better understood with reference to the following description and the appended claims. The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate exemplary embodiments of the disclosure and, together with the description, serve to explain the relevant principles. Attached Figure Description

[0011] This specification provides a detailed discussion of embodiments that are suitable for those skilled in the art, and the specification refers to the accompanying drawings, in which:

[0012] Figure 1A A block diagram of an example computational system for multimodal speech recognition, based on the examples described in this disclosure, is depicted.

[0013] Figure 1B A block diagram of an example computing device for performing multimodal speech recognition according to the examples described in this disclosure is depicted.

[0014] Figure 1C A block diagram of an example computing apparatus for training a machine learning multimodal speech recognition model according to the example described in this disclosure is depicted.

[0015] Figure 2 A block diagram depicts an example machine learning multimodal speech recognition model based on the examples described in this disclosure.

[0016] Figure 3A block diagram depicts an example machine learning multimodal speech recognition encoder / decoder model based on the examples described in this disclosure.

[0017] Figure 4 A data flow diagram depicts an example machine learning multimodal speech recognition model based on the examples described in this disclosure.

[0018] Figure 5 A data flow diagram of an example bimodal attention layer of a machine learning multimodal speech recognition model according to the examples described in this disclosure is depicted.

[0019] Figure 6 A data flow diagram depicts an example bimodal convolutional layer and an example bimodal pooling layer of a machine learning multimodal speech recognition model according to the examples described in this disclosure.

[0020] Figure 7 A flowchart depicts an example method for training a machine learning multimodal speech recognition model based on the examples described in this disclosure.

[0021] Repeated reference numerals on multiple figures are intended to identify the same features in various embodiments. Detailed Implementation

[0022] Overview

[0023] Generally, this disclosure relates to speech recognition. More specifically, this disclosure relates to training and utilizing a multimodal machine learning speech recognition model. As an example, the multimodal machine learning speech recognition model can be configured to operate in two modes: a streaming recognition mode or a context recognition mode. The streaming recognition mode can be configured to iteratively encode portions of speech data to obtain streaming prediction data (e.g., online encoding of phrases included in the speech data word by word while speaking / streaming the speech data). On the other hand, the context recognition mode can be configured to simultaneously encode portions of speech data to obtain context prediction data (e.g., offline encoding of all portions of the speech data simultaneously). The machine learning multimodal speech recognition model can encode speech data using either mode and can switch between modes based on the speech recognition task associated with the speech data (e.g., using the streaming recognition mode for virtual assistant tasks, using the context recognition mode for video captioning tasks, etc.). In this way, the machine learning multimodal speech recognition model can be used for both streaming speech recognition and contextual speech recognition, thus eliminating the need to design, train, and utilize separate models for various speech recognition modes.

[0024] In some embodiments, speech data may be or otherwise include spectral data (e.g., obtained using a spectrometer). Alternatively or additionally, in some embodiments, speech data may be or otherwise include audio data (e.g., phonemes, raw waveforms, raw audio, etc.). For example, speech data may be raw audio captured using an audio capture device. For another example, speech data may be raw waveform data collected via a waveform capture device. Thus, it should be noted that speech data may be, include, or be based on any data associated with the collection and / or analysis of speech.

[0025] Additionally, in some implementations, the two modes of the machine learning multimodal speech recognition model can be trained in a manner that distills the performance of the context recognition mode to further optimize the streaming recognition mode. As an example, speech data can be obtained along with associated ground truth labels (e.g., ground truth text transcriptions of speech included in the speech data). The context recognition mode of the machine learning multimodal speech recognition model can be used to process the speech data to obtain contextual prediction data (e.g., predictions that fully consider each part of the speech data). A contextual loss function can be evaluated, assessing the difference between the contextual prediction data and the ground truth labels. Next, the streaming recognition mode of the machine learning multimodal speech recognition model can be used to process the speech data to obtain streaming prediction data (e.g., predictions that iteratively consider each part of the speech data). A streaming loss function can be evaluated, assessing the difference between the streaming prediction data and the ground truth labels. Furthermore, a streaming loss function can be used to evaluate the difference between the contextual prediction data and the streaming prediction data (e.g., distillation loss based on more accurate processing of the context recognition mode). The parameters of the model can be tuned based on the two loss functions. In this way, the high-precision prediction of the context-aware pattern can be used as a distillation training signal, thus improving both the accuracy of the streaming pattern recognition and the transmission delay.

[0026] Specifically, in some examples, a loss function can be evaluated that assesses both a context loss term and a streaming loss term. For instance, the streaming loss term can assess the difference between the streaming predicted data and the ground truth labels, and the context loss term can assess the difference between the context-predicted data and the ground truth labels. The model's parameters can be tuned based on the loss function. Additionally, in some implementations, the loss function may include a distillation term. The distillation term assesses the difference between the context-predicted data and the streaming distilled data. In this way, the context-accurate prediction precision of the context-recognition pattern can be distilled to the streaming recognition pattern.

[0027] More specifically, the computing system may include a machine learning multimodal speech recognition model. The machine learning multimodal speech recognition model (e.g., encoders, decoders, etc.) may be or otherwise include one or more neural networks (e.g., deep neural networks). The neural network (e.g., deep neural network) may be a feedforward neural network, a convolutional neural network, a recurrent neural network (e.g., RNN-T, etc.), and / or various other types of neural networks. As an example, the machine learning multimodal speech recognition model may include multiple machine learning layers. These layers may include pointwise operators (e.g., feedforward networks, residual connections, activation layers, stepping layers, de-escalation layers, etc.), convolutional layers, recurrent neural network layers, long short-term memory layers, average pooling layers, self-attention layers, transformer layers, and / or normalization layers.

[0028] Machine learning multimodal speech recognition models can be configured to operate in either streaming or context-aware modes. A streaming mode can be configured to iteratively encode each of one or more portions of speech data to obtain streaming prediction data (e.g., processing speech data causally). As an example, a streaming mode can iteratively encode speech data in an autoregressive manner, where each prediction for the current portion of the speech data is conditioned on predictions for previous portions (e.g., when processing a portion of the speech data, future speech portions / context are not considered).

[0029] Context recognition patterns can be configured to simultaneously encode each of one or more portions of speech data to obtain context prediction data. More specifically, context recognition patterns can process each portion of speech data simultaneously, such that processing of any speech portion can influence the processing of another speech portion (e.g., processing speech data causally). As an example, context recognition patterns can simultaneously encode speech data in parallel, where each prediction of the current portion of the speech data is conditioned on predictions of previous and future portions (e.g., when processing a portion of speech data, both past and future speech portions / contexts are considered).

[0030] It should be noted that in some implementations, the machine learning multimodal speech recognition model may be the encoder portion of the overall model. As an example, the machine learning multimodal speech recognition model may be the encoder of a machine learning model and generate encodings that include, or otherwise include, speech prediction data (e.g., streaming prediction data, contextual prediction data, etc.). The decoder of the machine learning model may process the speech prediction data to obtain speech recognition output. Alternatively, in some implementations, the machine learning multimodal speech recognition model may be or otherwise include an encoder-decoder architecture.

[0031] In some implementations, each layer of a machine learning multimodal speech recognition model can be a streaming layer or a bimodal layer. A streaming layer can be configured to operate in streaming recognition mode, and a bimodal layer can be configured to operate in either streaming recognition mode or context recognition mode. It should be noted that because streaming layers do not operate in either mode, they do need to be adjusted or reconfigured when switching modes of the machine learning multimodal speech recognition model. More specifically, the functionality of streaming layers may be unaware of the model's current configuration mode and therefore can remain static regardless of the model's current mode.

[0032] In some implementations, the dual-mode layer can be configured to operate in either a streaming recognition mode (e.g., a non-causal mode) or a context recognition mode (e.g., a non-causal mode). As an example, each of the machine learning multimodal speech recognition model and the dual-mode layer can be configured to operate in streaming recognition mode. The machine learning multimodal speech recognition model can determine the context recognition mode as the selected mode (e.g., based on a speech recognition task associated with additional speech data). In response, each of the dual-mode layers can switch from streaming recognition mode to context recognition mode to facilitate context recognition using the machine learning multimodal speech recognition model.

[0033] As an example, at least one bimodal layer can be, or otherwise includes, a bimodal convolutional layer. To support the bimodal nature of the convolutional layer (e.g., streaming mode and context mode with shared weights), a normal convolution with a kernel size of k can be constructed, which can be applied in context recognition mode. Next, before applying the kernel in streaming recognition mode, a causal convolution with a kernel size of (k+1) / 2 can be simulated by constructing a boolean mask and multiplying it with a fully contextual convolutional kernel. The bimodal convolutional layer can include (k-1) / 2 additional parameters to support the fully contextual convolution (k). In contrast, a streaming mode convolutional layer can include ((k+1) / 2) parameters. Although this can introduce parameter overhead compared to normalized convolutional layers, it should be noted that in convolution-based models, these time-series convolutional layers account for only a very small amount of the total model size (e.g., most weights reside on a 1×1 convolutional layer with pointwise operators). Therefore, any parameter overhead introduced by utilizing bimodal convolutional layers can be considered negligible.

[0034] As another example, at least one dual-mode layer may be or otherwise include a dual-mode pooling layer. A dual-mode pooling layer may include one or more nodes (e.g., a sequential stack of average pooling (via time) layers, feedforward layers, activation layers, another feedforward layer, element-wise multiplication layers, etc.). To support both modes, the connections between one or more nodes can be adjusted. As an example, the connection between the first node (e.g., a feedforward layer, etc.) and subsequent convolutional layers can be adjusted to instead connect to different nodes in the dual-mode pooling layer. It should be noted that in some implementations, the dual-mode average pooling layer may be parameterless, thus avoiding the introduction of additional model size. In some implementations, the dual-mode pooling layer can be trained in parallel in streaming mode and can be readily implemented (e.g., using the "cumsum" function in TensorFlow, PyTorch, etc.).

[0035] As another example, at least one bimodal layer may be, or otherwise include, a bimodal attention layer. Self-attention (e.g., internal attention, etc.) can be viewed as an attention mechanism that weights different positions of a single sequence to compute representations of the same sequence. Furthermore, the attention layer itself is parameterless and may consist of matrix multiplications of the key and the query. In some implementations, the multiplication followed by the key may be a softmax normalization function processed by the key, prior to another matrix multiplication with the value. In streaming recognition mode, the bimodal attention layer may normalize only the "remaining" context (e.g., softmax, etc.) (e.g., compute self-attention only from portions of the already processed speech data). Conversely, in contextual recognition mode, normalization may be performed across the entire context of the speech data (e.g., compute self-attention from each portion of the speech data).

[0036] As another example, at least one bimodal layer may be, or otherwise include, a bimodal normalization layer. A streaming normalization function and a contextual normalization function can be instantiated for the bimodal normalization layer. Based on the patterns of a machine learning multimodal speech recognition model, one of the two normalization functions can be used. In this way, the bimodal normalization layer can switch between normalization functions based on the switching of patterns in the machine learning multimodal speech recognition model (e.g., switching from a batchnorm function to a layernorm function, etc.).

[0037] It should be noted that any pointwise operator can be or otherwise included as a bimodal layer. Pointwise operators can be or otherwise included in a neural network layer connecting input and output neurons at each time step (e.g., excluding cross-connections between different time steps). For example, these layers can include feedforward layers, fully connected layers, 1×1 convolutional layers, activation layers (e.g., ReLU, Swish, residual and dense connections, stepping layers, de-escalation layers, element-wise multiplication, etc.). Additionally, some of these layers can be or otherwise included in the convolutional transformer portion of a machine learning multimodal speech recognition model. For example, the convolutional transformer portion can include several bimodal layers, such as the first half of a feedforward layer, a multi-head self-attention layer, a convolutional layer, the second half of a feedforward layer, etc.

[0038] The computational system can obtain speech data and associated ground truth labels. Speech data can be training data and can include multiple parts (e.g., speech data segments). As an example, each part of the speech data can include one or more language units (e.g., phonemes, morphemes, words, letters, etc.). Alternatively, speech data can be assigned based on time (e.g., multiple lexical units corresponding to various time periods). Furthermore, the ground truth labels associated with the speech data can be textual representations of any speech included in the speech data.

[0039] Speech data can be processed using context recognition patterns from machine learning multimodal speech recognition models to obtain context prediction data. Context prediction data can be, or otherwise includes, predicted text transcriptions of the speech included in the speech data. For example, speech data may include statements spoken by a user to a virtual assistant application, and context prediction data may include predicted text transcriptions of those statements. It should be noted that context prediction data can be based on a non-causal (e.g., full context) understanding of the statements. More specifically, processing of each portion of the speech data can be at least partially based on any other portion of the speech data, as previously described.

[0040] A context loss function can be evaluated, which assesses the difference between the context-predicted data and the true labels. The context loss function can be, or otherwise includes, various loss functions such as mean squared error, likelihood loss, cross-entropy loss, hinge loss, and / or various other loss functions.

[0041] Speech data can be processed using streaming recognition patterns of machine learning multimodal speech recognition models to obtain streaming prediction data. Streaming prediction data can be, or otherwise include, predicted text transcriptions of speech included in the speech data in the same manner as context prediction data. It should be noted that context prediction data can be based on causal (e.g., non-contextual) understanding of the statements. More specifically, processing of each portion of the speech data can be based solely on the already processed portions of the speech data, as previously described.

[0042] A streaming loss function can be evaluated, which assesses the difference between the streaming predicted data and the true labels. Additionally, the streaming loss function can assess the difference between the streaming predicted data and the context-predicted data. The streaming loss function can be, or otherwise includes, various loss functions such as mean squared error, likelihood loss, cross-entropy loss, hinge loss, and / or various other loss functions.

[0043] One or more parameters of a machine learning multimodal speech recognition model can be tuned, at least in part, based on a context loss function and a streaming loss function. For example, two loss functions can be backpropagated through the model to update one or more parameters of the model (e.g., based on the gradients of the loss functions). Gradient descent techniques can be used to iteratively update the parameters over multiple training iterations. Alternatively, in some implementations, the two loss functions can be processed to generate a final loss (e.g., summed, averaged, concatenated, etc.), which can then be used to tune the model's parameters.

[0044] More specifically, to train a machine learning multimodal speech recognition model given a batch of data in each training iteration, in some implementations, one of the two model modes can be randomly sampled for training. Alternatively, in some implementations, both modes can be trained, and the two losses can be aggregated. In the former approach, known as random sampling training, the importance of the streaming recognition mode and the context recognition mode can be controlled by setting different sampling probabilities during training. In the latter approach, known as joint training, the importance can also be controlled to balance the streaming recognition and context recognition modes by assigning different loss weights.

[0045] Furthermore, as previously described, knowledge from context-aware patterns (e.g., the "teacher" pattern) can be dynamically distilled into streaming patterns (e.g., the "student" pattern) within the same general model (e.g., in-situ knowledge distillation). This distillation can be accomplished by utilizing a streaming loss function to evaluate the differences between context-predicted data and streaming-predicted data (e.g., encouraging consistency in predicted lexical probabilities). Since each iteration typically requires computation of predictions in both patterns, "teacher" predictions (e.g., context-predicted data) can be utilized without any additional computational or memory costs. This efficient knowledge distillation can be used in three parts P label P blank and 1-P label -P blank The probability is the KL divergence between the intrinsic context-aware pattern (e.g., "teacher") and the streaming pattern (e.g., "student"). Typically, predictions from context-aware patterns have lower latency because the pattern has no incentive to delay its output. Therefore, predictions generated by context-aware patterns may not necessarily be temporally aligned with streaming patterns. Consequently, temporal relaxation can be utilized by shifting the predictions of context-aware patterns one or more frames to the right, thus improving quality and training stability.

[0046] In some implementations, the computing system can acquire additional speech data corresponding to a speech recognition task (e.g., transcription task, closed captioning task, virtual assistant task, security recognition task, speech analysis task, bandwidth-constrained task, latency-constrained task, speech recognition task, etc.). Based on the speech recognition task, a selected mode can be determined from streaming recognition modes and context recognition modes of a machine learning multimodal speech recognition model. As an example, a speech recognition task can be associated with speech instructions received at a virtual assistant device. Based on the task, the selected mode can be determined as a streaming recognition mode. As an example, a speech recognition task can be associated with offline transcription of a newly released film for subtitling. Based on the task, the selected mode can be determined as a context recognition mode. The selected mode of the machine learning multimodal speech recognition model can be used to process the additional speech data to obtain speech recognition output.

[0047] In some implementations, the computing system can perform one or more actions, at least in part, based on the speech recognition output received. As an example, if the speech recognition output includes the transcription of instructions to a virtual assistant application, the virtual assistant application can perform or be instructed to transcribe the instructions (e.g., add a calendar entry, return the current time or weather, etc.). As another example, if the speech recognition output includes subtitles to an audio file associated with a video, the computing system can provide the subtitles to a video display application so that the subtitles can be overlaid on top of the video. Thus, the computing system can perform any action associated with the speech recognition task to facilitate its completion.

[0048] In some implementations, the proposed speech recognition technology can be provided as a cloud service for various users of a cloud-based machine learning platform. For example, a user can provision or otherwise identify one or more speech data examples (e.g., streaming speech data, context-aware speech data, etc.). The processes described herein can then be performed to automatically perform speech recognition on the user-identified or provided example speech data. For example, speech recognition can be combined with model training as a service. The trained model can be provided as output to the user (e.g., automatically deployed or deployed to one or more devices associated with the user).

[0049] This disclosure provides several technical effects and benefits. As an example of a technical effect and benefit, the systems and methods of this disclosure enable the use of both streaming speech recognition and contextual speech recognition technologies with a single multimodal speech recognition model. Both streaming speech recognition and contextual speech recognition technologies are extensively utilized in several cutting-edge technologies (e.g., human-computer understanding, speech recognition, semantic understanding, transcription, etc.). Consequently, the design, training, storage, and utilization of models for each mode have been previously required. However, this disclosure eliminates the need for using two or more models for various speech recognition contexts, thus significantly reducing the substantial computational costs (e.g., power, memory, storage devices, processing time, resource utilization, etc.) associated with designing, training, validating, and storing additional models for speech recognition. Furthermore, the bimodal nature reduces the complexity inherent in utilizing two separate models (e.g., the deployment, maintenance, monitoring, etc. of two separate models), thus significantly increasing user-friendliness and reducing the overhead associated with utilizing two models.

[0050] As another example of technical effect and benefit, the systems and methods disclosed herein achieve further improvements in streaming speech recognition. For example, by jointly training a machine learning multimodal speech recognition model in context and streaming recognition modes, the relatively high accuracy of the context recognition mode can be used as a distillation training signal for the streaming recognition mode. In this way, the performance of the streaming recognition mode (e.g., emission delay, word accuracy, etc.) can be improved without any additional processing costs (e.g., using a separate model as a distillation training signal, etc.). These improvements to streaming recognition performance can reduce the number of erroneous word predictions, thus saving several resources associated with additional processing or incorrect predictions (e.g., memory usage, network bandwidth usage, etc.).

[0051] Example devices and systems

[0052] Figure 1A A block diagram of an example computing system 100 for performing multimodal speech recognition is depicted. System 100 includes a user computing device 102, a server computing system 130, and a training computing system 150 communicatively coupled via a network 180.

[0053] User computing device 102 can be any type of computing device, such as, for example, a personal computing device (e.g., a laptop computer or desktop computer), a mobile computing device (e.g., a smartphone or tablet), a game console or controller, a wearable computing device, an embedded computing device, or any other type of computing device.

[0054] User computing device 102 includes one or more processors 112 and memory 114. The one or more processors 112 can be any suitable processing device (e.g., processor core, microprocessor, ASIC, FPGA, controller, microcontroller, etc.) and can be a single processor or multiple processors operatively connected. Memory 114 can include one or more non-transitory computer-readable storage media, such as RAM, ROM, EEPROM, EPROM, flash memory devices, disks, etc., and combinations thereof. Memory 114 can store data 116 and instructions 118 that, when executed by processor 112, cause user computing device 102 to operate.

[0055] In some embodiments, the user computing device 102 may store or include one or more machine learning multimodal speech recognition models 120. For example, the machine learning multimodal speech recognition model 120 may be, or may otherwise include, various machine learning models, such as neural networks (e.g., deep neural networks) or other types of machine learning models, including nonlinear and / or linear models. Neural networks may include feedforward neural networks, recurrent neural networks (e.g., long short-term memory recurrent neural networks), convolutional neural networks, or other forms of neural networks. Refer to Figures 1 to... Figure 3 Example machine learning multimodal speech recognition model 120 is discussed.

[0056] In some implementations, one or more machine learning multimodal speech recognition models 120 may be received from server computing system 130 via network 180, stored in user computing device memory 114, and then used or otherwise implemented by one or more processors 112. In some implementations, user computing device 102 may implement multiple parallel instances of a single machine learning multimodal speech recognition model 120 (e.g., parallel machine learning multimodal speech recognition across multiple instances of the machine learning multimodal speech recognition model).

[0057] More specifically, the machine learning multimodal speech recognition model 120 can be configured to operate in two modes: a streaming recognition mode or a context recognition mode. The streaming recognition mode can be configured to encode portions of speech data individually and / or iteratively to obtain streaming prediction data (e.g., online encoding of phrases included in the speech data word by word while speaking / streaming the speech data). On the other hand, the context recognition mode can be configured to encode portions of speech data simultaneously to obtain context prediction data (e.g., offline encoding of all portions of the speech data simultaneously). The machine learning multimodal speech recognition model 120 can encode speech data using either mode and can switch between modes based on the speech recognition task associated with the speech data (e.g., using the streaming recognition mode for virtual assistant tasks, using the context recognition mode for video captioning tasks, etc.). In this way, the machine learning multimodal speech recognition model 120 can be used for both streaming speech recognition and contextual speech recognition, thus eliminating the need to design, train, and utilize separate models for various speech recognition modes.

[0058] Alternatively, one or more machine learning multimodal speech recognition models 140 may be included in or otherwise stored and implemented by server computing system 130, which communicates with user computing device 102 according to a client-server relationship. For example, machine learning multimodal speech recognition model 140 may be implemented by server computing system 140 as part of a web service (e.g., a speech recognition service). Therefore, one or more models 120 may be stored and implemented at user computing device 102 and / or one or more models 140 may be stored and implemented at server computing system 130.

[0059] User computing device 102 may also include one or more user input components 122 for receiving user input. For example, user input component 122 may be a touch-sensitive component (e.g., a touch-sensitive display or touchpad) that is sensitive to the touch of a user input object (e.g., a finger or stylus). Touch-sensitive components may be used to implement a virtual keyboard. Other example user input components include a microphone, a conventional keyboard, or other components that a user may use to provide user input.

[0060] Server computing system 130 includes one or more processors 132 and memory 134. The one or more processors 132 can be any suitable processing device (e.g., processor core, microprocessor, ASIC, FPGA, controller, microcontroller, etc.) and can be a single processor or multiple processors operatively connected. Memory 134 can include one or more non-transitory computer-readable storage media, such as RAM, ROM, EEPROM, EPROM, flash memory devices, disks, etc., and combinations thereof. Memory 134 can store data 136 and instructions 138 that, when executed by processor 132, cause server computing system 130 to operate.

[0061] In some embodiments, the server computing system 130 includes one or more server computing devices or is otherwise implemented by one or more server computing devices. Where the server computing system 130 includes multiple server computing devices, these server computing devices may operate according to a sequential computing architecture, a parallel computing architecture, or some combination thereof.

[0062] As described above, the server computing system 130 may store or otherwise include one or more machine learning multimodal speech recognition models 140. For example, model 140 may be, or may otherwise include, various machine learning models. Example machine learning models include neural networks or other multi-layered nonlinear models. Example neural networks include feedforward neural networks, deep neural networks, recurrent neural networks, and convolutional neural networks. Refer to Figures 1 to... Figure 3Example model 140 was discussed.

[0063] User computing device 102 and / or server computing system 130 can train models 120 and / or 140 via interaction with training computing system 150, which is communicatively coupled to network 180. Training computing system 150 may be separate from server computing system 130 or may be part of server computing system 130.

[0064] The training computing system 150 includes one or more processors 152 and memory 154. The one or more processors 152 can be any suitable processing device (e.g., processor core, microprocessor, ASIC, FPGA, controller, microcontroller, etc.) and can be a single processor or multiple processors operatively connected. The memory 154 can include one or more non-transitory computer-readable storage media, such as RAM, ROM, EEPROM, EPROM, flash memory devices, disks, etc., and combinations thereof. The memory 154 can store data 156 and instructions 158 that, when executed by the processor 152, cause the training computing system 150 to operate. In some embodiments, the training computing system 150 includes one or more server computing devices or is otherwise implemented by one or more server computing devices.

[0065] The training computing system 150 may include a model trainer 160 that uses various training or learning techniques, such as, for example, backpropagation of error, to train machine learning models 120 and / or 140 stored at the user computing device 102 and / or the server computing system 130. For example, a loss function can be backpropagated through the model to update one or more parameters of the model (e.g., gradients based on the loss function). Various loss functions can be used, such as mean squared error, likelihood loss, cross-entropy loss, hinge loss, and / or various other loss functions. Gradient descent techniques can be used to iteratively update parameters over multiple training iterations.

[0066] In some implementations, backpropagation of the error may include performing backpropagation via time truncation. The model trainer 160 may perform several generalization techniques (e.g., weight decay, loss of weight, etc.) to improve the generalization ability of the trained model.

[0067] Specifically, model trainer 160 can train machine learning multimodal speech recognition models 120 and / or 140 based on a set of training data 162. Training data 162 may include, for example, speech data and associated ground truth labels. Speech data may be training data and may include multiple parts (e.g., speech data segments, etc.). As an example, each part of the speech data may include one or more language units (e.g., phonemes, word elements, words, letters, etc.). Alternatively or additionally, speech data may be assigned based on time (e.g., multiple lexical units corresponding to various time periods, etc.). Furthermore, the ground truth labels associated with the speech data may be text representations of any speech included in the speech data.

[0068] In some implementations, training examples can be provided by the user computing device 102 if the user has already provided consent. Therefore, in this implementation, the model 120 provided to the user computing device 102 can be trained by training the computing system 150 on user-specific data received from the user computing device 102. In some cases, this process may be referred to as personalizing the model.

[0069] Model trainer 160 includes computer logic for providing desired functionality. Model trainer 160 can be implemented using hardware, firmware, and / or software that controls a general-purpose processor. For example, in some embodiments, model trainer 160 includes a program file stored on a storage device, loaded into memory, and executed by one or more processors. In other embodiments, model trainer 160 includes one or more sets of computer-executable instructions stored in a tangible computer-readable storage medium (such as RAM, hard disk, optical media, or magnetic media).

[0070] Network 180 can be any type of communication network, such as a local area network (e.g., intranet), a wide area network (e.g., the Internet), or some combination thereof, and can include any number of wired or wireless links. Typically, communication via network 180 can be conducted using various communication protocols (e.g., TCP / IP, HTTP, SMTP, FTP), encodings or formats (e.g., HTML, XML), and / or protection schemes (e.g., VPN, Secure HTTP, SSL) via any type of wired and / or wireless connection.

[0071] The machine learning models described in this specification can be used in a variety of tasks, applications, and / or use cases.

[0072] In some implementations, the input to the machine learning model of this disclosure can be text or natural language data. The machine learning model can process the text or natural language data to generate output. As an example, the machine learning model can process natural language data to generate language-encoded output. As another example, the machine learning model can process text or natural language data to generate latent text embedding output. As another example, the machine learning model can process text or natural language data to generate translation output. As another example, the machine learning model can process text or natural language data to generate classification output. As another example, the machine learning model can process text or natural language data to generate text segmentation output. As another example, the machine learning model can process text or natural language data to generate semantic intent output. As another example, the machine learning model can process text or natural language data to generate improved quality text or natural language output (e.g., text or natural language data of higher quality than the input text or natural language). As another example, the machine learning model can process text or natural language data to generate predictive output.

[0073] In some embodiments, the input to the machine learning model of this disclosure can be speech data. The machine learning model can process the speech data to generate output. As an example, the machine learning model can process speech data to generate speech recognition output. As another example, the machine learning model can process speech data to generate speech translation output. As another example, the machine learning model can process speech data to generate latent embedding output. As another example, the machine learning model can process speech data to generate encoded speech output (e.g., encoded and / or compressed representation of speech data). As another example, the machine learning model can process speech data to generate improved quality speech output (e.g., speech data of higher quality than the input speech data). As another example, the machine learning model can process speech data to generate text representation output (e.g., text representation of the input speech data). As another example, the machine learning model can process speech data to generate predictive output.

[0074] In some implementations, the input to the machine learning model of this disclosure can be latent encoded data (e.g., a latent spatial representation of the input). The machine learning model can process the latent encoded data to generate an output. As an example, the machine learning model can process the latent encoded data to generate a recognition output. As another example, the machine learning model can process the latent encoded data to generate a reconstruction output. As another example, the machine learning model can process the latent encoded data to generate a search output. As another example, the machine learning model can process the latent encoded data to generate a re-clustering output. As yet another example, the machine learning model can process the latent encoded data to generate a prediction output.

[0075] In some embodiments, the input to the machine learning model of this disclosure can be sensor data. The machine learning model can process the sensor data to generate output. As an example, the machine learning model can process sensor data to generate a recognition output. As another example, the machine learning model can process sensor data to generate a prediction output. As another example, the machine learning model can process sensor data to generate a classification output. As another example, the machine learning model can process sensor data to generate a segmentation output. As another example, the machine learning model can process sensor data to generate a segmentation output. As another example, the machine learning model can process sensor data to generate a visualization output. As another example, the machine learning model can process sensor data to generate a diagnostic output. As another example, the machine learning model can process sensor data to generate a detection output.

[0076] In some cases, the input includes audio data representing spoken utterances, and the task is a speech recognition task. The output may include text output mapped to spoken utterances.

[0077] Figure 1A The illustration shows an example computing system that can be used to implement this disclosure. Other computing systems may also be used. For example, in some embodiments, user computing device 102 may include a model trainer 160 and a training dataset 162. In this embodiment, model 120 can be trained and used locally at user computing device 102. In some such embodiments, user computing device 102 may implement model trainer 160 to personalize model 120 based on user-specific data.

[0078] Figure 1B A block diagram of an example computing device 10 for performing multimodal speech recognition is depicted. The computing device 10 may be a user computing device or a server computing device.

[0079] The computing device 10 includes several applications (e.g., applications 1 to N). Each application contains its own machine learning library and machine learning model. For example, each application may include a machine learning model. Example applications include text messaging applications, email applications, dictation applications, virtual keyboard applications, browser applications, etc.

[0080] like Figure 1B As illustrated, each application can communicate with several other components of the computing device, such as, for example, one or more sensors, a context manager, a device state component, and / or additional components. In some implementations, each application can communicate with each device component using an API (e.g., a public API). In some implementations, the API used by each application is application-specific.

[0081] Figure 1CA block diagram depicts an example computing device 50 for training a machine learning multimodal speech recognition model. The computing device 50 can be a user computing device or a server computing device.

[0082] The computing device 50 includes several applications (e.g., applications 1 to N). Each application communicates with a central intelligence layer. Example applications include text messaging applications, email applications, dictation applications, virtual keyboard applications, browser applications, etc. In some implementations, each application may communicate with the central intelligence layer (and the models stored therein) using an API (e.g., a common API across all applications).

[0083] The central intelligence layer includes several machine learning models. For example, such as... Figure 1C As illustrated, a corresponding machine learning model (e.g., a model) can be provided for each application and managed by a central intelligence layer. In other embodiments, two or more applications can share a single machine learning model. For example, in some embodiments, the central intelligence layer can provide a single model (e.g., a single model) for all applications. In some embodiments, the central intelligence layer is included within the operating system of computing device 50 or otherwise implemented by the operating system of computing device 50.

[0084] The central intelligence layer can communicate with the central device data layer. The central device data layer can be a centralized storage repository for data used by the computing device 50. For example... Figure 1C As illustrated, the central device data layer can communicate with several other components of the computing device, such as, for example, one or more sensors, a context manager, a device state component, and / or additional components. In some implementations, the central device data layer can communicate with each device component using an API (e.g., a proprietary API).

[0085] Example model layout

[0086] Figure 2A block diagram of an example machine learning multimodal speech recognition model 200 is depicted. In some embodiments, the machine learning multimodal speech recognition model 200 is trained to receive a set of input data 204 describing speech data, and as a result of receiving the input data 204, provides transcribed input data 204 or output data 206 encoded thereon in other ways. As an example, the machine learning multimodal speech recognition model 200 can be configured to operate in two modes: streaming recognition mode or context recognition mode. Streaming recognition mode can be configured to encode portions of speech data 204 individually and / or iteratively to obtain streaming prediction data (e.g., online encoding of phrases included in the speech data word by word while speaking / streaming the speech data). On the other hand, context recognition mode 204 can be configured to jointly and / or simultaneously encode portions of speech data to obtain context prediction data (e.g., offline encoding of all portions of the speech data simultaneously). The machine learning multimodal speech recognition model 200 can encode speech data 204 using any mode and can switch between modes based on the speech recognition task associated with the speech data 204 (e.g., using streaming recognition mode to handle virtual assistant tasks, using context recognition mode to handle video captioning tasks, etc.). Based on the machine learning multimodal speech recognition model 200, output data 206 describing, instructing, and / or transcribing the content of input data 204 can be obtained.

[0087] Figure 3 A block diagram of an example machine learning multimodal speech recognition encoder / decoder model 300 is depicted. Except that the machine learning multimodal speech recognition model 300 further includes a machine learning multimodal speech recognition encoder 202 and a machine learning multimodal speech recognition decoder 302, the machine learning multimodal speech recognition model 300 is similar to... Figure 2 A machine learning multimodal speech recognition model 200 is described. As an example, a machine learning multimodal speech recognition encoder 202 may be an encoder of the machine learning multimodal speech recognition model 300 and may generate encoding 304, which is or otherwise includes speech prediction data (e.g., streaming prediction data, context prediction data, etc.). A machine learning multimodal speech recognition decoder 302 may be a decoder portion of the machine learning multimodal speech recognition model. The machine learning multimodal speech recognition decoder 302 may process the speech prediction data 304 to obtain output data 206, which is or otherwise includes speech recognition output.

[0088] Figure 4A data flow diagram of an example machine learning multimodal speech recognition model 400 is depicted. As an example, the modern streaming recognition mode 402A and context recognition mode 402B of speech recognition models 402 / 404 share most of the neural architecture and training recipe, with the most significant difference being in the speech recognition encoder 402. Typically, the streaming recognition mode 402A of the neural network speech recognition encoder 402 is an autoregressive model, where each prediction at the current time step is conditioned on previous predictions (without future context). In contrast, the context recognition mode of the speech recognition encoder 402 can be fully contextual and non-causal.

[0089] Figure 5 A data flow diagram depicts an example bimodal attention layer 500 of a machine learning multimodal speech recognition model. As depicted, the bimodal attention layer 500 may include a normalization function 502. Switching the mode of the machine learning multimodal speech recognition model from context recognition mode 504A to streaming recognition mode 504B may include masking one or more inputs to the normalization function. For example, by switching from mode 504A to 504B, one or more of the inputs are masked by the masking softmax normalization function 502.

[0090] Figure 6 The data flow diagrams depict example bimodal convolutional layer 600A and example bimodal pooling layer 600B of the machine learning multimodal speech recognition model are described.

[0091] As an example, to support the dual-mode nature of the dual-mode convolutional layer 600A (e.g., streaming mode and context mode with shared weights), a normal convolution with a kernel size of k can be constructed, which can be applied in context recognition mode. Next, before applying the kernel in streaming recognition mode, a causal convolution with a kernel size of (k+1) / 2 can be simulated by constructing a boolean mask and multiplying it by the dual-mode convolutional layer 600A. The dual-mode convolutional layer 600A can include (k-1) / 2 additional parameters to support a fully contextual convolution (k). In contrast, a streaming mode convolutional layer can include (k+1) / 2 parameters. Although this introduces parameter overhead compared to normalized convolutional layers, it should be noted that in convolution-based models, these time-series convolutional layers account for only a very small amount of the total model size (e.g., most weights reside on 1×1 convolutional layers, which are pointwise operators, etc.). Therefore, any parameter overhead introduced by utilizing the dual-mode convolutional layer 600A can be considered negligible.

[0092] As another example, the dual-mode pooling layer 600B may include one or more nodes (e.g., a sequential stack of average pooling (via time) layers, feedforward layers, activation layers, another feedforward layer, element-wise multiplication layers, etc.). To support both modes, the connections between one or more nodes of the dual-mode pooling layer 600B can be adjusted. As an example, instead, the connection between the first node (e.g., a feedforward layer, etc.) and subsequent convolutional layers (e.g., layer 600A) can be adjusted to connect to different nodes of the dual-mode pooling layer 600B. It should be noted that in some implementations, the dual-mode pooling layer 600B may be parameterless, thus avoiding the introduction of additional model size. In some implementations, the dual-mode pooling layer 600B can be trained in parallel in streaming mode and can be easily implemented (e.g., using the "cumsum" function in TensorFlow, PyTorch, etc.).

[0093] Example Method

[0094] Figure 7 A flowchart depicts an example method 700 for training a machine learning multimodal speech recognition model. Although for illustrative and discussion purposes... Figure 7 The steps are described in a specific order, but the method of this disclosure is not limited to the specific order or arrangement described. Various steps of method 700 may be omitted, rearranged, combined, and / or adjusted in various ways without departing from the scope of this disclosure. Furthermore, it should be understood that although... Figure 7 This paper presents an example method for training a multimodal speech recognition model, but other methods are also possible. The use of the multimodal speech recognition model described herein is not based on training... Figure 7 The model described in the example is based on the model.

[0095] In section 702, the computational system can acquire speech data. More specifically, the computational system can acquire speech data and associated ground truth labels. Speech data can be training data and can include multiple parts (e.g., speech data segments, etc.). As an example, each part of the speech data can include one or more linguistic units (e.g., phonemes, glyphs, words, letters, etc.). Alternatively or additionally, speech data can be assigned based on time (e.g., multiple lexical units corresponding to various time periods, etc.). Furthermore, the ground truth labels associated with the speech data can be text representations of any speech included in the speech data.

[0096] In section 704, a computational system may use a machine learning multimodal speech recognition model to process speech data to obtain contextual prediction data. More specifically, the computational system may use a contextual recognition pattern of a machine learning multimodal speech recognition model to process speech data to obtain contextual prediction data. Contextual prediction data may be, or otherwise include, predicted text transcriptions of the speech included in the speech data. For example, speech data may include statements spoken by a user to a virtual assistant application, and contextual prediction data may include predicted text transcriptions of those statements. It should be noted that contextual prediction data may be based on a non-causal (e.g., full context) understanding of the statements. More specifically, processing of each part of the speech data may be at least partially based on any other part of the speech data.

[0097] More specifically, the computing system may include a machine learning multimodal speech recognition model. The machine learning multimodal speech recognition model (e.g., encoders, decoders, etc.) may be or otherwise include one or more neural networks (e.g., deep neural networks). The neural network (e.g., deep neural network) may be a feedforward neural network, a convolutional neural network, a recurrent neural network (e.g., RNN-T, etc.), and / or various other types of neural networks. As an example, the machine learning multimodal speech recognition model may include multiple machine learning layers. These layers may include pointwise operators (e.g., feedforward networks, residual connections, activation layers, stepping layers, de-escalation layers, etc.), convolutional layers, recurrent neural network layers, long short-term memory layers, average pooling layers, self-attention layers, transformer layers, and / or normalization layers.

[0098] Machine learning multimodal speech recognition models can be configured to operate in either a streaming recognition mode or a contextual recognition mode. A streaming recognition mode can be configured to encode each of one or more portions of speech data individually and / or iteratively to obtain streaming prediction data (e.g., processing speech data in a non-causal manner). As an example, a streaming recognition mode can encode speech data individually and / or iteratively in an autoregressive manner, where each prediction of the current portion of the speech data is conditioned on predictions of previous portions (e.g., when processing a portion of the speech data, future speech portions / context are not considered).

[0099] Context recognition patterns can be configured to jointly and / or simultaneously encode each of one or more portions of speech data to obtain context prediction data. More specifically, context recognition patterns can process each portion of speech data simultaneously, such that processing of any speech portion can influence the processing of another speech portion (e.g., processing speech data causally). As an example, context recognition patterns can jointly and / or simultaneously encode speech data in parallel, where each prediction of the current portion of the speech data is conditioned on predictions of previous and future portions (e.g., when processing a portion of speech data, both past and future speech portions / contexts are considered, etc.).

[0100] It should be noted that in some implementations, the machine learning multimodal speech recognition model may be the encoder portion of the overall model. As an example, the machine learning multimodal speech recognition model may be the encoder of a machine learning model and generate encodings that include, or otherwise include, speech prediction data (e.g., streaming prediction data, contextual prediction data, etc.). The decoder of the machine learning model may process the speech prediction data to obtain speech recognition output. Alternatively, in some implementations, the machine learning multimodal speech recognition model may be or otherwise include an encoder-decoder architecture.

[0101] In some implementations, each layer of a machine learning multimodal speech recognition model can be a streaming layer or a bimodal layer. A streaming layer can be configured to operate in streaming recognition mode, and a bimodal layer can be configured to operate in either streaming recognition mode or context recognition mode. It should be noted that because streaming layers do not operate in either mode, they do need to be adjusted or reconfigured when switching modes of the machine learning multimodal speech recognition model. More specifically, the functionality of streaming layers may be unaware of the model's current configuration mode and therefore can remain static regardless of the model's current mode.

[0102] In some implementations, the dual-mode layer can be configured to operate in either a streaming recognition mode (e.g., a non-causal mode) or a context recognition mode (e.g., a non-causal mode). As an example, each of the machine learning multimodal speech recognition model and the dual-mode layer can be configured to operate in streaming recognition mode. The machine learning multimodal speech recognition model can determine the context recognition mode as the selected mode (e.g., based on a speech recognition task associated with additional speech data). In response, each of the dual-mode layers can switch from streaming recognition mode to context recognition mode to facilitate context recognition using the machine learning multimodal speech recognition model.

[0103] As an example, at least one bimodal layer can be, or otherwise includes, a bimodal convolutional layer. To support the bimodal nature of the convolutional layer (e.g., streaming mode and context mode with shared weights), a normal convolution with a kernel size of k can be constructed, which can be applied in context recognition mode. Next, before applying the kernel in streaming recognition mode, a causal convolution with a kernel size of (k+1) / 2 can be simulated by constructing a boolean mask and multiplying it with a full-context convolutional kernel. The bimodal convolutional layer can include (k-1) / 2 additional parameters to support the full-context convolution (k). In contrast, a streaming mode convolutional layer can include (k+1) / 2 parameters. Although this can introduce parameter overhead compared to normalized convolutional layers, it should be noted that in convolution-based models, these time-series convolutional layers account for only a very small amount of the total model size (e.g., most weights reside on 1×1 convolutional layers, which are pointwise operators, etc.). Therefore, any parameter overhead introduced by utilizing bimodal convolutional layers can be considered negligible.

[0104] As another example, at least one dual-mode layer may be or otherwise include a dual-mode pooling layer. A dual-mode pooling layer may include one or more nodes (e.g., a sequential stack of average pooling (via time) layers, feedforward layers, activation layers, another feedforward layer, element-wise multiplication layers, etc.). To support both modes, the connections between one or more nodes can be adjusted. As an example, the connection between the first node (e.g., a feedforward layer, etc.) and subsequent convolutional layers can be adjusted to instead connect to different nodes in the dual-mode pooling layer. It should be noted that in some implementations, the dual-mode average pooling layer may be parameterless, thus avoiding the introduction of additional model size. In some implementations, the dual-mode pooling layer can be trained in parallel in streaming mode and can be readily implemented (e.g., using the "cumsum" function in TensorFlow, PyTorch, etc.).

[0105] As another example, at least one bimodal layer may be, or otherwise include, a bimodal attention layer. Self-attention (e.g., internal attention, etc.) can be viewed as an attention mechanism that weights different positions of a single sequence to compute representations of the same sequence. Furthermore, the attention layer itself is parameterless and may consist of matrix multiplications of the key and the query. In some implementations, the multiplication followed by the key may be a softmax normalization function processed by the key, prior to another matrix multiplication with the value. In streaming recognition mode, the bimodal attention layer may normalize only the "remaining" context (e.g., softmax, etc.) (e.g., compute self-attention only from the already processed portions of the speech data). Conversely, in context recognition mode, normalization may be performed across the entire context of the speech data (e.g., compute self-attention from each portion of the speech data).

[0106] As another example, at least one bimodal layer may be, or otherwise include, a bimodal normalization layer. A streaming normalization function and a contextual normalization function can be instantiated for the bimodal normalization layer. Based on the patterns of a machine learning multimodal speech recognition model, one of the two normalization functions can be used. In this way, the bimodal normalization layer can switch between normalization functions based on the switching of patterns in the machine learning multimodal speech recognition model (e.g., switching from a batchnorm function to a layernorm function, etc.).

[0107] It should be noted that any pointwise operator can be or otherwise included as a bimodal layer. Pointwise operators can be or otherwise included in a neural network layer connecting input and output neurons at each time step (e.g., excluding cross-connections between different time steps). For example, these layers can include feedforward layers, fully connected layers, 1×1 convolutional layers, activation layers (e.g., ReLU, Swish, residual and dense connections, stepping layers, de-escalation layers, element-wise multiplication, etc.). Additionally, some of these layers can be part of a convolutional transformer portion of a machine learning multimodal speech recognition model or otherwise included within that convolutional transformer portion. For example, the convolutional transformer portion can include several bimodal layers, such as the first half of a feedforward layer, a multi-head self-attention layer, a convolutional layer, the second half of a feedforward layer, etc.

[0108] In 706, the computational system can evaluate a context loss function. More specifically, the computational system can evaluate a context loss function that assesses the difference between the context-predicted data and the true label. The context loss function can be, or otherwise includes, various loss functions such as mean squared error, likelihood loss, cross-entropy loss, hinge loss, and / or various other loss functions.

[0109] In 708, the computational system can use a machine learning multimodal speech recognition model to process speech data to obtain streaming prediction data. More specifically, the computational system can use a streaming recognition pattern of the machine learning multimodal speech recognition model to process speech data to obtain streaming prediction data. Streaming prediction data can be, or otherwise includes, predicted text transcriptions of speech included in the speech data in the same manner as context prediction data. It should be noted that context prediction data can be based on causal (e.g., non-contextual) understanding of the statements. More specifically, processing of each portion of the speech data can be based solely on the already processed portions of the speech data, as previously described.

[0110] In 710, the computational system can evaluate a streaming loss function. More specifically, the computational system can evaluate a streaming loss function that assesses the difference between the streaming predicted data and the true value labels. Additionally, the streaming loss function can assess the difference between the streaming predicted data and the context-predicted data. The streaming loss function can be, or otherwise includes, various loss functions such as mean squared error, likelihood loss, cross-entropy loss, hinge loss, and / or various other loss functions.

[0111] In 712, the computational system can tune the parameters of a machine learning multimodal speech recognition model. More specifically, the computational system can tune one or more parameters of the machine learning multimodal speech recognition model, at least in part, based on a context loss function and a streaming loss function. For example, the two loss functions can be backpropagated through the model to update one or more parameters of the model (e.g., based on the gradients of the loss functions). Gradient descent techniques can be used to iteratively update the parameters over multiple training iterations. Alternatively, in some implementations, the two loss functions can be processed to generate a final loss (e.g., summation, averaging, concatenation, etc.), which can then be used to tune the model's parameters.

[0112] More specifically, to train a machine learning multimodal speech recognition model given a batch of data in each training iteration, in some implementations, one of the two model modes can be randomly sampled for training. Alternatively, in some implementations, both modes can be trained, and the two losses can be aggregated. In the former approach, known as random sampling training, the importance of the streaming recognition mode and the context recognition mode can be controlled by setting different sampling probabilities during training. In the latter approach, known as joint training, the importance can also be controlled to balance the streaming recognition and context recognition modes by assigning different loss weights.

[0113] Furthermore, as previously described, knowledge from context-aware patterns (e.g., “teacher” patterns) can be dynamically distilled into streaming patterns (e.g., “student” patterns) within the same general model (e.g., in-situ knowledge distillation). This distillation can be accomplished by evaluating the difference between context-predicted data and streaming-predicted data using a streaming loss function (e.g., encouraging consistency in predicted lexical probabilities). Since each iteration typically requires computation of predictions in both patterns, “teacher” predictions (e.g., context-predicted data) can be utilized without any additional computational or memory costs. This efficient knowledge distillation can be used in three parts: P label P blank and 1-P label -P blankThe probabilities are expressed as the KL divergence between context-aware patterns (e.g., "teacher") and streaming patterns (e.g., "student"). Typically, predictions from context-aware patterns have lower latency because the pattern has no incentive to delay its output. Therefore, predictions generated by context-aware patterns may not necessarily be temporally aligned with streaming patterns. Consequently, temporal relaxation can be utilized by right-shifting the predictions of context-aware patterns by one or more frames, thus improving quality and training stability.

[0114] In some implementations, the computing system can acquire additional speech data corresponding to a speech recognition task (e.g., transcription task, closed captioning task, virtual assistant task, security recognition task, speech analysis task, bandwidth-constrained task, latency-constrained task, speech recognition task, etc.). Based on the speech recognition task, a selected mode can be determined from streaming recognition modes and context recognition modes of a machine learning multimodal speech recognition model. As an example, a speech recognition task can be associated with speech instructions received at a virtual assistant device. Based on the task, the selected mode can be determined as a streaming recognition mode. As an example, a speech recognition task can be associated with offline transcription of a newly released film for subtitling. Based on the task, the selected mode can be determined as a context recognition mode. The selected mode of the machine learning multimodal speech recognition model can be used to process the additional speech data to obtain speech recognition output.

[0115] In some implementations, the computing system can perform one or more actions, at least in part, based on the speech recognition output received. As an example, if the speech recognition output includes the transcription of instructions to a virtual assistant application, the virtual assistant application can perform or be instructed to transcribe the instructions (e.g., add a calendar entry, return the current time or weather, etc.). As another example, if the speech recognition output includes subtitles to an audio file associated with a video, the computing system can provide the subtitles to a video display application so that the subtitles can be overlaid on top of the video. Thus, the computing system can perform any action associated with the speech recognition task to facilitate its completion.

[0116] Additional Disclosure

[0117] This document discusses technical reference servers, databases, software applications, and other computer-based systems, as well as the actions taken and the information sent to and from such systems. The inherent flexibility of computer-based systems allows for a wide range of possible configurations, combinations, and divisions of tasks and functionalities between and within components. For example, the processes discussed herein can be implemented using a single device or component, or multiple devices or components working in combination. Databases and applications can be implemented on a single system or distributed across multiple systems. Distributed components can operate sequentially or in parallel.

[0118] While the subject matter has been described in detail with respect to various specific example embodiments, each example is provided by way of illustration and not limitation. Modifications, variations, and equivalents to such embodiments will be readily apparent to those skilled in the art upon gaining an understanding of the foregoing. Therefore, this disclosure does not preclude such modifications, variations, and / or additions to the subject matter as will be readily apparent to those of ordinary skill in the art. For example, features illustrated or described as part of an embodiment may be used with another embodiment to produce yet another embodiment. Therefore, this disclosure is intended to cover such modifications, variations, and equivalents.

Claims

1. A computer-implemented method for utilizing a multimodal speech recognition model, comprising: A machine learning multimodal speech recognition model is obtained by a computing system including one or more computing devices. The machine learning multimodal speech recognition model is configured to operate in both a streaming recognition mode and a context recognition mode. The machine learning multimodal speech recognition model is configured to switch between the streaming recognition mode and the context recognition mode. The streaming recognition mode is configured to iteratively encode each of one or more portions of speech data, and the context recognition mode is configured to simultaneously encode each of the one or more portions of speech data. The computing system obtains speech data corresponding to the speech recognition task; The computing system determines the selected mode from the streaming recognition mode and the context recognition mode of the machine learning multimodal speech recognition model based on the speech recognition task; as well as The computing system processes the speech data using the selected mode of the machine learning multimodal speech recognition model to obtain speech recognition output.

2. The computer-implemented method of claim 1, wherein the machine learning multimodal speech recognition model comprises a plurality of layers, wherein each layer comprises a streaming layer configured to operate in the streaming recognition mode or a bimodal layer configured to operate selectively in the streaming recognition mode and the context recognition mode.

3. The computer-implemented method of claim 2, wherein at least one dual-mode layer includes a dual-mode convolutional layer, the dual-mode convolutional layer including a convolutional kernel, and wherein switching the mode includes applying a mask to the convolutional kernel.

4. The computer-implemented method of claim 1, wherein the selected mode includes the context recognition mode, and wherein the speech recognition task includes: Transcription task; Hidden subtitles task; or Verbal analysis task.

5. The computer-implemented method of claim 1, wherein the selected mode includes the streaming recognition mode, and wherein the speech recognition task includes: Bandwidth-constrained tasks; Delayed constraint tasks; Speech recognition task; or Virtual assistant task.

6. The computer-implemented method according to claim 1, wherein the method further comprises: In response to obtaining the speech recognition output, the computing system performs one or more actions based at least in part on the speech recognition output.

7. A computational implementation method for training machine learning models for multimodal speech recognition: The machine learning multimodal speech recognition model is configured to operate in both streaming recognition mode and context recognition mode, wherein the machine learning multimodal speech recognition model is configured to switch between the streaming recognition mode and the context recognition mode, wherein the streaming recognition mode is configured to iteratively encode each of one or more portions of the speech data to obtain streaming prediction data, and wherein the context recognition mode is configured to simultaneously encode each of the one or more portions of the speech data to obtain context prediction data. The method includes: Obtain the speech data and associated ground truth labels; The speech data is processed using the context recognition pattern of the machine learning multimodal speech recognition model to obtain context prediction data; Evaluate the context loss function, which assesses the difference between the context-predicted data and the true label. The speech data is processed using the streaming recognition mode of the machine learning multimodal speech recognition model to obtain streaming prediction data; Evaluate the streaming loss function, which assesses the difference between the streaming predicted data and the true value labels; as well as One or more parameters of the machine learning multimodal speech recognition model are adjusted, at least in part, based on the context loss function and the streaming loss function.

8. The method of claim 7, wherein the streaming loss function further evaluates the difference between the context prediction data and the streaming prediction data.

9. The method of claim 8, wherein evaluating the context loss function and the streaming loss function comprises: The evaluation comprises a loss term consisting of multiple items, wherein said multiple loss terms include: A context loss term, which evaluates the difference between the context-predicted data and the true value label; A streaming loss term, which evaluates the difference between the streaming predicted data and the true value labels; and A distillation loss term, which evaluates the difference between the contextual prediction data and the streaming prediction data.

10. The method of claim 7, wherein the machine learning multimodal speech recognition model comprises a plurality of layers, wherein each layer comprises a streaming layer configured to operate in the streaming recognition mode or a bimodal layer configured to operate selectively in the streaming recognition mode and the context recognition mode.

11. The method of claim 10, wherein before processing the speech data using the streaming recognition pattern, the method comprises: Switch one or more modes in the dual-mode layer from the streaming recognition mode to the context recognition mode.

12. The method of claim 11, wherein at least one dual-mode layer comprises a dual-mode convolutional layer, the dual-mode convolutional layer comprising a convolutional kernel, and wherein switching the mode comprises applying a mask to the convolutional kernel.

13. The method of claim 11, wherein at least one dual-mode layer includes a dual-mode pooling layer, the dual-mode pooling layer comprising one or more nodes, and wherein switching the mode comprises: Adjust the connection of at least one of the one or more nodes to the corresponding node in the adjacent layer of the plurality of layers.

14. The method of claim 11, wherein at least one bimodal layer comprises a bimodal attention layer, the bimodal attention layer comprising a normalization function, and wherein switching the modes comprises: Mask one or more inputs to the normalization function.

15. The method of claim 11, wherein at least one bimodal layer includes a bimodal normalization layer, the bimodal normalization layer including a streaming normalization function and a context normalization function, wherein the bimodal normalization layer is configured to utilize the context normalization function, and wherein switching the mode includes: The dual-mode normalization layer is reconfigured to utilize the streaming normalization function.

16. The method of claim 11, wherein a subset of the plurality of layers includes a convolution transformer portion, the convolution transformer portion comprising: The first half of the feedforward layer; Multi-head self-attention layer; Convolutional layer; as well as The second half of the feedforward layer.

17. The method of claim 7, wherein the method further comprises: Obtain additional speech data corresponding to the speech recognition task; Based on the speech recognition task, determine the selected mode from the streaming recognition mode and the context recognition mode of the machine learning multimodal speech recognition model; as well as The additional speech data is processed using the selected mode of the machine learning multimodal speech recognition model to obtain speech recognition output.

18. The method of claim 17, wherein the speech recognition task comprises: Transcription task; Hidden subtitles task; or Virtual assistant task.

19. One or more tangible, non-transitory computer-readable media storing computer-readable instructions, which, when executed by one or more processors, cause the one or more processors to perform operations, the operations including: A machine learning multimodal speech recognition model is obtained, the machine learning multimodal speech recognition model being configured to operate in both streaming recognition mode and context recognition mode, wherein the machine learning multimodal speech recognition model is configured to switch between the streaming recognition mode and the context recognition mode, wherein the streaming recognition mode is configured to iteratively encode each of one or more portions of the speech data, and wherein the context recognition mode is configured to simultaneously encode each of the one or more portions of the speech data. Obtain speech data corresponding to the speech recognition task; Based on the speech recognition task, determine the selected mode from the streaming recognition mode and the context recognition mode of the machine learning multimodal speech recognition model; as well as The speech data is processed using the selected mode of the machine learning multimodal speech recognition model to obtain speech recognition output.

20. The one or more tangible, non-transient media of claim 19, wherein the selected mode includes the context recognition mode, and wherein the speech recognition task includes: Transcription task; Hidden subtitles task; or Verbal analysis task.

21. The one or more tangible, non-transient media of claim 19, wherein the selected mode includes the streaming recognition mode, and wherein the speech recognition task includes: Bandwidth-constrained tasks; Delayed constraint tasks; Speech recognition task; or Virtual assistant task.

Images