Speech decoding applications of multimodal machine learning

Abstract

Systems and techniques for speech synthesis and analysis are illustrated. One embodiment includes a method that obtains a plurality of recordings corresponding to at least one vocalization, wherein each recording of the plurality of recordings includes a sensor measurement in a unique data modality that is obtained from a speaker during the at least one vocalization. The method encodes the plurality of recordings, using at least one encoder, into a shared latent space, wherein: each of the plurality of recordings includes a plurality of down-sampled timesteps where each corresponds to a distinct latent embedding. The method aligns the shared latent space, wherein, when the plurality of recordings is obtained concurrently: aligning the shared latent space includes temporally aligning the distinct latent embeddings across the plurality of encoded recordings using a cross-contrastive loss function. The method transduces the aligned latent space into synthesized speech.

Description

Speech Decoding Applications of Multimodal Machine LearningCROSS-REFERENCE TO RELATED APPLICATIONS

[0001] The current application claims the benefit of and priority under 35 U.S.C. § 119(e) to U.S. Provisional Patent Application No. 63 / 588,649, entitled “Systems and Methods for Multimodal Machine Learning in Speech Decoding,” filed October 6, 2023, and U.S. Provisional Patent Application No. 63 / 649,181 , entitled “Systems and Methods for Multimodal Machine Learning in Speech Decoding,” filed May 17, 2024, the disclosures of which are hereby incorporated by reference in their entireties for all purposes.FIELD OF THE INVENTION

[0002] The present invention generally relates to automatic speech recognition (ASR) systems and, more specifically, training such systems for decoding diverse forms of speech.BACKGROUND

[0003] Speech recognition processes describe configurations dedicated to identifying, deciphering, and / or interpreting human speech. A common system for speech recognition involves the audio processing of input vocalizations. Examples of this include speech-to-text applications, generally directed to converting spoken language to its written form. This has a variety of potential applications such as real-time dictation, support for hearing-impaired individuals, and producing translations between various languages. Speech synthesis is an effective inversion of standard (audio-based) speech recognition processes, referring to the artificial production or reconstruction of human speech (for example, text-to-speech applications). Silent Speech Interfaces (SSIs) represent a branch of human-computer interaction that offers non-invasive means of nonverbal communication. SSIs can facilitate speech-based communication without having to depend on audible vocalizations. SSIs may be used for human-computer interaction, particularly for individuals with speech impairments and in scenarios where traditional vocal communication is impractical. In doing so, SSIs effectively implement pseudospeech recognition by estimating the intended vocalizations of specific speakers, using speech synthesis to emulate the estimated vocalizations.SUMMARY OF THE INVENTION

[0004] Systems and techniques for speech synthesis and analysis are illustrated. One embodiment includes a method for decoding attempted speech. The method obtains a plurality of recordings corresponding to at least one vocalization, wherein each recording of the plurality of recordings includes a sensor measurement in a unique data modality that is obtained from a speaker during the at least one vocalization. The method encodes the plurality of recordings, using at least one encoder, into a shared latent space, wherein: each of the plurality of recordings includes a plurality of down-sampled timesteps; and each down-sampled timestep of the plurality of down-sampled timesteps corresponds to a distinct latent embedding. The method aligns the shared latent space, wherein, when the plurality of recordings is obtained concurrently: aligning the shared latent space includes temporally aligning the distinct latent embeddings across the plurality of encoded recordings; and temporally aligning the distinct latent embeddings across the plurality of encoded recordings is performed using a cross-contrastive loss function. The method transduces the aligned latent space into synthesized speech.

[0005] In a further embodiment, when temporally aligning the distinct latent embeddings across the plurality of encoded recordings using the cross-contrastive loss function, similarity for the distinct latent embeddings is maximized for similar timesteps and minimized for different timesteps.

[0006] In another embodiment, when the plurality of recordings is obtained asynchronously: aligning the shared latent space includes phonemically aligning the distinct latent embeddings for the plurality of encoded recordings; and phonemically aligning the distinct latent embeddings for the plurality of encoded recordings is performed using a supervised temporal contrastive loss function.

[0007] In a further embodiment, when phonemically aligning the distinct latent embeddings across the plurality of encoded recordings using the cross-contrastive loss function, similarity for the distinct latent embedding is maximized for similar phoneme class labels and minimized for different phoneme class labels.

[0008] In a still further embodiment, the phoneme class labels distinguish between phonemes and silence.

[0009] In another further embodiment, phonemically aligning the distinct latent embeddings for the plurality of encoded recordings is performed, at least in part, using a softmax function.

[0010] In yet another further embodiment, the similarity for the distinct latent embedding is determined, at least in part, using dynamic time warping.

[0011] In another embodiment, each of the plurality of down-sampled timesteps represents a temporal offset between: a first recording of the plurality of recordings; and a second recording of the plurality of recordings.

[0012] In a further embodiment, the first recording of the plurality of recordings includes an Electromyography (EMG) measurement; and the second recording of the plurality of recordings includes an audio measurement.

[0013] In another embodiment, the method trains a decoder to decode silent speech using the aligned latent space.

[0014] In a further embodiment, an audio-only dataset is used to train at least one of: the at least one encoder; or the decoder.

[0015] In another embodiment, the at least one encoder includes at least one of an artificial neural network (ANN), and a recurrent neural network (RNN), or a specialized neural network.

[0016] In yet another embodiment, the method trains the at least one encoder using at least one of: Connectionist Temporal Classification (CTC) loss values generated based on aligning the shared latent space; or the aligned shared latent space.

[0017] In still yet another embodiment, the method improves recognition accuracy of the synthesized speech using at least one of: a heuristic search algorithm; or a large language model.

[0018] In another embodiment, the unique data modality of at least one of the plurality of recordings is selected from the group consisting of brain implant measurements, lip reading measurements, ultrasound measurements, magnetic resonance imaging (MRI) measurements, Functional Near-Infrared Spectroscopy (fNIRS) measurements, magnetoencephalography (MEG) measurements,electroencephalogram (EEG) measurements, radar measurements, strain sensor measurements, and non-audible murmur measurements.

[0019] One embodiment includes a non-transitory computer-readable medium including instructions that, when executed, are configured to cause a processor to perform a process for decoding attempted speech. The processor obtains a plurality of recordings corresponding to at least one vocalization, wherein each recording of the plurality of recordings includes a sensor measurement in a unique data modality that is obtained from a speaker during the at least one vocalization. The processor encodes the plurality of recordings, using at least one encoder, into a shared latent space, wherein: each of the plurality of recordings includes a plurality of down-sampled timesteps; and each down-sampled timestep of the plurality of down-sampled timesteps corresponds to a distinct latent embedding. The processor aligns the shared latent space, wherein, when the plurality of recordings is obtained concurrently: aligning the shared latent space includes temporally aligning the distinct latent embeddings across the plurality of encoded recordings; and temporally aligning the distinct latent embeddings across the plurality of encoded recordings is performed using a cross-contrastive loss function. The processor transduces the aligned latent space into synthesized speech.

[0020] In a further embodiment, when temporally aligning the distinct latent embeddings across the plurality of encoded recordings using the cross-contrastive loss function, similarity for the distinct latent embeddings is maximized for similar timesteps and minimized for different timesteps.

[0021] In another embodiment, when the plurality of recordings is obtained asynchronously: aligning the shared latent space includes phonemically aligning the distinct latent embeddings for the plurality of encoded recordings; and phonemically aligning the distinct latent embeddings for the plurality of encoded recordings is performed using a supervised temporal contrastive loss function.

[0022] In a further embodiment, when phonemically aligning the distinct latent embeddings across the plurality of encoded recordings using the cross-contrastive loss function, similarity for the distinct latent embedding is maximized for similar phoneme class labels and minimized for different phoneme class labels.

[0023] In a still further embodiment, the phoneme class labels distinguish between phonemes and silence.

[0024] In another further embodiment, phonemically aligning the distinct latent embeddings for the plurality of encoded recordings is performed, at least in part, using a softmax function.

[0025] In yet another further embodiment, the similarity for the distinct latent embedding is determined, at least in part, using dynamic time warping.

[0026] In another embodiment, each of the plurality of down-sampled timesteps represents a temporal offset between: a first recording of the plurality of recordings; and a second recording of the plurality of recordings.

[0027] In a further embodiment, the first recording of the plurality of recordings includes an Electromyography (EMG) measurement; and the second recording of the plurality of recordings includes an audio measurement.

[0028] In another embodiment, the processor trains a decoder to decode silent speech using the aligned latent space.

[0029] In a further embodiment, an audio-only dataset is used to train at least one of: the at least one encoder; or the decoder.

[0030] In another embodiment, the at least one encoder includes at least one of an artificial neural network (ANN), and a recurrent neural network (RNN), or a specialized neural network.

[0031] In yet another embodiment, the processor trains the at least one encoder using at least one of: Connectionist Temporal Classification (CTC) loss values generated based on aligning the shared latent space; or the aligned shared latent space.

[0032] In still yet another embodiment, the processor improves recognition accuracy of the synthesized speech using at least one of: a heuristic search algorithm; or a large language model.

[0033] In another embodiment, the unique data modality of at least one of the plurality of recordings is selected from the group consisting of brain implant measurements, lip reading measurements, ultrasound measurements, magnetic resonance imaging (MRI) measurements, Functional Near-Infrared Spectroscopy (fNIRS) measurements, magnetoencephalography (MEG) measurements,electroencephalogram (EEG) measurements, radar measurements, strain sensor measurements, and non-audible murmur measurements.

[0034] One embodiment includes a system for decoding attempted speech, including: at least one sensor; a network interface; at least one encoder; and a processor configured to execute instructions. The processor obtains a plurality of recordings corresponding to at least one vocalization, wherein each recording of the plurality of recordings includes a sensor measurement, measured by one or more sensors of the at least one sensor, in a unique data modality that is obtained from a speaker during the at least one vocalization. The processor encodes the plurality of recordings, using at least one encoder, into a shared latent space, wherein: each of the plurality of recordings includes a plurality of down-sampled timesteps; and each down-sampled timestep of the plurality of down-sampled timesteps corresponds to a distinct latent embedding. The processor aligns the shared latent space, wherein, when the plurality of recordings is obtained concurrently: aligning the shared latent space includes temporally aligning the distinct latent embeddings across the plurality of encoded recordings; and temporally aligning the distinct latent embeddings across the plurality of encoded recordings is performed using a cross-contrastive loss function. The processor transduces the aligned latent space into synthesized speech.

[0035] In a further embodiment, when temporally aligning the distinct latent embeddings across the plurality of encoded recordings using the cross-contrastive loss function, similarity for the distinct latent embeddings is maximized for similar timesteps and minimized for different timesteps.

[0036] In another embodiment, when the plurality of recordings is obtained asynchronously: aligning the shared latent space includes phonemically aligning the distinct latent embeddings for the plurality of encoded recordings; and phonemically aligning the distinct latent embeddings for the plurality of encoded recordings is performed using a supervised temporal contrastive loss function.

[0037] In a further embodiment, when phonemically aligning the distinct latent embeddings across the plurality of encoded recordings using the cross-contrastive loss function, similarity for the distinct latent embedding is maximized for similar phoneme class labels and minimized for different phoneme class labels.

[0038] In a still further embodiment, the phoneme class labels distinguish between phonemes and silence.

[0039] In another further embodiment, phonemically aligning the distinct latent embeddings for the plurality of encoded recordings is performed, at least in part, using a softmax function.

[0040] In yet another further embodiment, the similarity for the distinct latent embedding is determined, at least in part, using dynamic time warping.

[0041] In another embodiment, each of the plurality of down-sampled timesteps represents a temporal offset between: a first recording of the plurality of recordings; and a second recording of the plurality of recordings.

[0042] In a further embodiment, the first recording of the plurality of recordings includes an Electromyography (EMG) measurement; and the second recording of the plurality of recordings includes an audio measurement.

[0043] In another embodiment, the processor trains a decoder to decode silent speech using the aligned latent space.

[0044] In a further embodiment, an audio-only dataset is used to train at least one of: the at least one encoder; or the decoder.

[0045] In another embodiment, the at least one encoder includes at least one of an artificial neural network (ANN), and a recurrent neural network (RNN), or a specialized neural network.

[0046] In yet another embodiment, the processor trains the at least one encoder using at least one of: Connectionist Temporal Classification (CTC) loss values generated based on aligning the shared latent space; or the aligned shared latent space.

[0047] In still yet another embodiment, the processor improves recognition accuracy of the synthesized speech using at least one of: a heuristic search algorithm; or a large language model.

[0048] In another embodiment, the unique data modality of at least one of the plurality of recordings is selected from the group consisting of brain implant measurements, lip reading measurements, ultrasound measurements, magnetic resonance imaging (MRI) measurements, Functional Near-Infrared Spectroscopy (fNIRS) measurements, magnetoencephalography (MEG) measurements,electroencephalogram (EEG) measurements, radar measurements, strain sensor measurements, and non-audible murmur measurements.

[0049] Additional embodiments and features are set forth in part in the description that follows, and in part will become apparent to those skilled in the art upon examination of the specification or may be learned by the practice of the invention. A further understanding of the nature and advantages of the present invention may be realized by reference to the remaining portions of the specification and the drawings, which form a part of this disclosure.BRIEF DESCRIPTION OF THE DRAWINGS

[0050] The description and claims will be more fully understood with reference to the following figures and data graphs, which are presented as exemplary embodiments of the invention and should not be construed as a complete recitation of the scope of the invention.

[0051] FIG. 1 illustrates a Multimodal Orofacial Neural Audio system in accordance with a number of embodiments of the invention.

[0052] FIGS. 2 - 3 conceptually illustrate latent space alignments performed in accordance with several embodiments of the invention.

[0053] FIGS. 4 - 5 conceptually illustrate a Silent Speech Recognition system implemented in accordance with multiple embodiments of the invention.DETAILED DESCRIPTION

[0054] Speech decoding refers to a class of technologies focused on extracting the substance of speech (e.g., words, phonemes) from non-audio signals. For example, speech has been decoded from neural signals recorded from a user’s brain associated with muscle movements in the face. Neural speech decoding systems can be particularly useful for users who have motor impairment, limited speech capabilities, and / or neurological defects. Similarly, electromyography (EMG) signals have been used to apply the muscular response to nerve stimulation to decode (i.e. , extrapolate likely intended) speech without the requirement of explicit user vocalizations. As suggested above, speech decoding has clear value in the medical space as it can restore the capacity of speech to individuals whose vocal capacity has been limited (e.g., due to injury or illness).However, speech decoding can enable silent speech outside of the medical field as well. For example, using speech decoding systems implemented in accordance with several embodiments of the invention, military personnel can communicate via speech-decoding modalities merely by having EMG sensors on the inside of their helmet. Civilians could equally find value in communicating without vocalizing by having their speech (or a transcription thereof) transmitted to other users for communication, or to connected devices as commands. Indeed, the use cases for silent speech are myriad. SSIs have historically faced challenges in implementation, such as the absence of phonetic information to achieve error rates suitable for practical use. An especially common problem with existing EMG-based speech decoding systems is that there is insufficient training data to adequately train current speech decoding machine learning models.

[0055] Configurations operating in accordance with various embodiments of the invention may be implemented to train multimodal (e.g., neural network) models applied to silent speech recognition and / or decoding. Configurations may include but are not limited to the Multimodal Orofacial Neural Audio (MONA) system which may be used to leverage alignments between any group of two or more data (e.g., speech) modalities. Data modalities that implementations of the MONA system can be used alongside may include but are not limited to measurements / recordings obtained from brain implant, lip reading, ultrasound, magnetic resonance imaging (MRI), Functional Near-Infrared Spectroscopy (fNIRS), magnetoencephalography (MEG), electroencephalogram (EEG), radar, strain sensor, non-audible murmur, and / or EMG measurements.

[0056] Systems and methods in accordance with several embodiments of the invention indicate that recordings of various data modalities (including but not limited to audio signals of speech and EMG signals related to speech generation) can share latent representations, thereby increasing the effectiveness they have in training multimodal models. In many embodiments, a method referred to herein as cross-contrastive loss can be used to construct shared latent space(s) for aligned audio speech signals and corresponding EMG signals. In some embodiments, a method referred to herein as supervised temporal contrastive loss can be used on unaligned data modalities (e.g., audio speech signals and EMG signals with different text).

[0057] MONA may compare the various data modalities through loss functions in order to produce the (shared) latent representations using incorporated encoders. The loss functions used in accordance with certain embodiments of the invention may include but are not limited to cross-contrast (crossCon) and supervised temporal contrast (supTcon) loss functions. When multiple data modalities are recorded concurrently, encoders may be trained, in accordance with a number of embodiments of the invention, by crossCon loss functions. In doing so, crossCon loss functions may pair (or group, for when more than two data modalities are involved) latent embeddings for simultaneously recorded measurements. As such, crossCon loss functions may attempt to classify paired latent representations as the same “class”, and / or non-paired latent representations as different classes. Additionally or alternatively, when data from multiple modalities are available, but not recorded concurrently, encoders may be trained using supTcon loss functions. These loss functions may attempt to classify all latent representations with the same phoneme class together and / or differentiate them from all latent representations with different phoneme classes. These two loss functions, one directed to temporal proximity, the other to phonemic similarity, may be used to encourage latent representations to be shared across data modalities. In many cases, performance can benefit the most when both techniques / functions are applied simultaneously.

[0058] Several embodiments of the invention may incorporate additional steps to refine silent speech recognition. For example, a post-processing step may follow heuristic search algorithms including but not limited to beam searches to synthesize the predicted sentence(s) from the top (e.g., top k) candidates. Additionally or alternatively, systems operating in accordance with several embodiments of the invention may implement Large Language Model (LLM) Integrated Scoring Adjustment (LISA) processes to improve recognition accuracy including but not limited to word error rate (WER) measurements. Such processes may be applied (but are not limited) to improving the accuracy of any of a variety of speech-to-text prediction models. Systems may (additionally or alternatively) utilize LLMs to capitalize on extensive text datasets to improve silent speech recognition. Notably, systems and methods in accordance with various embodiments of the invention may consistently achieve WERs below the critical 15% threshold, indicating the viabilityof SSIs implemented in accordance with several embodiments of the invention for openvocabulary applications.

[0059] Incorporating techniques operating in accordance with various embodiments of the invention, systems implemented in accordance with multiple embodiments of the invention may narrow performance gaps between silent and vocalized speech; open new possibilities in human-computer interaction; demonstrate the potential of cross-modal approaches in noisy and data-limited regimes; and / or demonstrate the effectiveness of SSIs as a viable alternative to automatic speech recognition (ASR). In some embodiments, multimodal models may harness audio-only datasets including but not limited to the LibriSpeech corpus. In such cases, models and / or systems may use LibriSpeech clean + other for training and / or LibriSpeech clean for validation and testing.

[0060] By incorporating additional data modalities, additional training data can be made accessible, including but not limited to (e.g., thousands of hours of) available speech-to-text datasets. Access to these datasets may significantly dwarf dataset sizes for EMG / neural datasets (which may only have tens of hours); and / or dataset sizes for video-to-text lipreading (which may only have hundreds of hours). Additionally or alternatively, training on data that includes information about speech, articulators, and / or neural representations, may allow artificial neural networks configured in accordance with numerous embodiments of the invention to be capable of learning more generalizable representations of the underlying audio signals in model-based approaches.A. Multimodal Orofacial Neural Audio Systems

[0061] A MONA system configuration used for training multimodal models configured in accordance with several embodiments of the invention is illustrated in FIG. 1 . Datasets used for training multimodal models configured in some embodiments of the invention may include but are not limited to the Gaddy Silent Speech EMG dataset. Additionally or alternatively, multimodal models in accordance with various embodiments of the invention may be implemented as neural network models including but not limited to Resnet -Transform er (ResNet) models.

[0062] In accordance with many embodiments of the invention, vocalized utterances associated with particular statements may be used to derive cross-modal recordings of data modalities, that may include but are not limited to EMG 110 and audio 120 recordings (as in FIG. 1 ). In accordance with some embodiments, EMGs may be used for cross-modal recordings on the basis that, when placed over facial muscles, EMG can reliably detect activity related to speech articulation and, when placed on the throat, can detect internal motion of the larynx and vocal cord. Additionally or alternatively, systems may rely on audio data from vocal utterances to enhance the decoding of silent speech. That said, other data modalities recorded in accordance with multiple embodiments of the invention may include but are not limited to neural microelectrode arrays, electrocochleography (ECoG) readings, camera imagery of speech articulators, wireless sensing reading, infrared results, and / or ultrasound results.

[0063] In accordance with many embodiments of the invention, recordings of data modalities are encoded using encoders 130, 135, that may include but are not limited to artificial neural networks (ANNs), into (unified) latent 140 representations. The ANNs implemented as encoders 130, 135 in accordance with multiple embodiments of the invention may include but are not limited to recurrent neural networks (RNNs), transformers, ResNets (as in FIG. 1 ), and specialized neural networks (including but not limited to S4 and Hyena).

[0064] In accordance with various embodiments of the invention, latent representations (or latent space 140) may be used to enhance the training of (universal) decoders 180 in decoding silent speech. In accordance with many embodiments of the invention, latent space 140 may correspond to individual (e.g., down-sampled) timesteps. In several embodiments of the invention, single decoders 180 can be trained to translate latent space 140 into text (i.e., [silent] speech-to-text). Additionally or alternatively, transducers 190 may be trained to transduce latent space 140 into synthesized audio and / or speech (e.g., text-to-speech synthesis). By incorporating additional data modalities, more training data can be used, such as the available speech-to-text datasets and / or video-to-text lipreading datasets. Additionally or alternatively, Connectionist Temporal Classification (CTC) loss 170 values for (e.g., aligning) individual data modalities may be used to further refine the effectiveness of the encoders 130, 135 basedon text 175 training datasets and / or letter logits 185 determined by the decoder(s) 180 during speech-to-text synthesis.

[0065] In accordance with certain embodiments of the invention, supTcon loss 155 (determined by supTcon loss functions) may be used for constructing shared representations in latent space 140 across unaligned data modalities, including but not limited to pairings of vocalized audio 120 recordings and (asynchronous) silent EMG 110 recordings. For scenarios where multiple data modalities are recorded, but not recorded concurrently, the encoders can be trained using supTcon loss 155 functions to classify all latent embeddings with the same (e.g. , previously determined) phoneme 160 class as the same, and different from all latent embeddings with different phoneme 160 classes. In accordance with many embodiments of the invention, class labels may be used per each latent embedding to align latent representations across data modalities and datasets. For example, the class labels may be particular phonemes and / or indications of silence as found with Montreal Forced Aligner.

[0066] In accordance with numerous embodiments of the invention, crossCon loss 150 (determined by crossCon loss functions) may be used for constructing shared representations in latent space across aligned data modalities. This may include but is not limited to pairings of vocalized audio 120 recordings and (aligned) EMG 110 recordings. When multiple data modalities are recorded concurrently, encoders configured in accordance with some embodiments of the invention can be trained using crossCon loss functions to classify paired latent embeddings as the same class and / or non-paired latent embeddings as different classes. Models incorporating the use of crossCon may train substantially faster than other models, corresponding to improved generalization.

[0067] FIG. 2 illustrates a latent space alignment, for time-aligned data, performed using a crossCon loss function implemented in accordance with various embodiments of the invention. Under MONA (performed using crossCon), given a vocalized utterance i with simultaneous recordings (e.g., EMG and audio recordings), the output of the encoder corresponding to a first data modality (in this case, the EMG recording) may take the form:with t timesteps (e.g., of 10 ms) each. The simultaneous output of the encoder corresponding to a second data modality (in this case, the audio recording) may be represented as:In such cases, VtVi(et i, at i~) may represent latent embeddings for the simultaneously recorded EMG and audio data. Systems may represent all latent embeddings for a particular minibatch with n total utterances by:where Z e 1RPXLfor F features and L total embeddings. To simplify indexing, systems can define function j i) to return the corresponding (e.g., EMG / audio) index for utterance i. Then, the crossCon loss function may be defined by Equation 1 for temperature T:Equation 2 may represent a negative log-likelihood (crossCon) loss, used to classify the specific positive pair of EMG & audio latent embeddings among a set of distractors.Equation 3 may represent the average loss from L classification problems. In obtaining this value, systems may source distractors from both domains of data.

[0068] Systems implemented in accordance with certain embodiments of the invention may have the task of creating encoders where cross-modality etiiand atiiare more similar than the same-modality et-i zand etii, and therefore encourages a shared latent representation for EMG and audio data. As such, systems may be configured so the similarity of latent embeddings for simultaneously recorded EMG and audio data is maximized for the same timestep, and / or minimized to other timesteps.

[0069] FIG. 3 illustrates a latent space alignment, for time-independent data, performed with supTcon in accordance with a number of embodiments of the invention. Under MONA implementations (performed using supTcon loss functions), given vocalized utterance i with simultaneous EMG and audio recordings, the output of the EMG encoder may take the form:with t timesteps (e.g., of 10ms) each, where the t timesteps may (additionally or alternatively) represent a temporal offset between the EMG and the audio data. The output for the audio encoder may be represented as:To simplify the indexing notation, implementations may introduce function p(i), defined to return the set of all indices corresponding to entries that share the same class label(phoneme) as the entry at index i. simfz zjSystems operating in accordance with several embodiments of the invention may perform summations over positives in Equation 4 outside of the log. Additionally or alternatively, in supTcon, the number of comparisons may be quadratic in the total number of timesteps across all examples in the utterance / batch, rather than quadratic in the number of examples per batch.

[0070] In accordance with some embodiments, supTcon may be especially effective when only silent EMG data are available. Here, we only consider supervised learning where text labels are available; however, supTcon might be extended to the semi-supervised learning case where only subsets of class labels are available (e.g., at least in part) through methods including but not limited to using softmax functions (e.g., Gumbel-Softmax) for phoneme classification, allowing the use of additional data.

[0071] While unique configurations for training models are described above with reference to FIGS. 1 - 3, they may incorporate various attributes as appropriate to therequirements of specific applications in accordance with many embodiments of the invention. Additionally, the specific manner in which MONA may be implemented in accordance with various embodiments of the invention is largely dependent upon the requirements of a given application.B. Silent Speech Recognition via Dynamic Time Warping

[0072] In accordance with multiple embodiments of the invention, supTcon and / or crossCon may be used for latent space alignment through the use of Dynamic Time Warping (DTW) for pursposes including but not limited to aligning distinct latent embeddings. Since there may not be usable audio data in silent speech conditions, systems in accordance with various embodiments of the invention may take advantage of (prior) parallel silent and / or vocalized utterances to align the silent condition to the parallel utterance(s) in the vocalized condition. Since DTW on EMG data can perform worse than on audio data, systems may temporally align the silent condition with the vocalized condition in the latent space. This can allow the encoders to act as denoisers and extract meaningful semantic content that may improve DTW results (i.e., the MONA approach). Let an EMG latent of silent utterance be Z s JRrixcwith T±timesteps and C channels, and let a parallel vocalized EMG recording of utterance be Z2E RAXC and phonemes P2E P7”2where P is set that includes phonemes and silence token with T2timesteps. Systems may first calculate the Euclidean distance matrix DThen, using DTW, systems can calculate the alignment of Z2onto Z±, and use this alignment to create the warped Z2E RAXCanc|2allowing for the calculation of crossCon and supTcon.

[0073] In order to calculate crossCon and / or supTcon with DTW, systems in accordance with multiple embodiments of the invention may incorporate at least one silent EMG example with parallel vocalized EMG and Audio per minibatch. As memory usage for contrastive loss functions is quadratic with input length, systems may (additionally or alternatively) sample batches below a maximum length to ensure sufficient GPU memory. In (common) cases where there is massively more audio-to-text data than EMG-to-text, systems may undersample LibriSpeech so that it accounts for approximately 50% of theexamples per epoch. The heuristic for solving this class-proportional bin packing problem(which can be used for Batch Samplers) is described in the following algorithm:Input: item lengths L, item class labels C, max bin length M, class proportions P, required classes per bin IOutput: list of bins with item indices BInitialize B and sum list sums to emptyGroup indices by class into idxShuffle idx per classInitialize debt for each class to 0 while num( / dx) > 0 for all classes doSample class c based on P if debt of c > 0 thenDecrement debt for cContinue end ifPop item with length t from L and class c from C if there exists a bin where { + sum(lengths in bin) < M thenFind the first such binAdd item idx to this bin and update sums elseCreate new bin; Add item idx to bin if I classes are missing thenAdd idx for missing I classes to new binIncrement debt for added classes end if end if end whileInitialize failure count f to 0Set failure threshold T while failure count < T and remaining items exist doAdjust to avoid sampling classes without itemsSample class c based on PTry to add item to existing bin without exceeding M if added successfully thenReset f elseIncrement f end if end whileDiscard any remaining items

[0074] Dynamic Time Warping implemented in accordance with many embodiments of the invention are not limited to use within SSIs. Accordingly, it should be appreciatedthat applications described herein may also be implemented outside the context of speech decoding. Moreover, while specific loss function configurations are described above, any of a variety of loss functions can be utilized alongside Dynamic Time Warping as appropriate to the requirements of specific applications.C. Large Language Model Integrated Scoring Adjustment & Additional Operations

[0075] During training, systems and methods in accordance with numerous embodiments of the invention may use 4-gram language models and / or beam sizes of at least 150 for beam searches. By increasing the beam size to 5000 for inference, systems can reduce WER significantly. In alternative or addition to calculating the posterior negative log-likelihood (NLL) from the beam search prior and updating with LLMs, systems may prompt Generative Pre-trained Transformer (GPTs) multimodal large language models including but not limited to GPT-3.5 (“gpt-3.5-turbo-16k-0613”) and / or GPT-4 (“gpt-4- 0125-preview”) with task descriptions and lists of the top predictions. These predictions can be sourced from the top k beams from beam searches and / or from ensembles of models (for which finetuning the ensembles on diverse datasets may further encourage task performance through ensemble weighting). Sourcing predictions on ensembles of models may, additionally or alternatively, have higher predicted text diversity as is depicted in the example below.Your task is to perform automatic speech recognition. Below are multiple candidate transcriptions, listed from most likely to least likely. Choose the transcription that is most accurate, ensuring it is contextually and grammatically correct. Focus on key differences in the options that change the meaning or correctness. Avoid selections with repetitive or nonsensical phrases. In cases of ambiguity, select the option that is most coherent and contextually sound. Respond with the chosen transcription only, without any introductory text. after breakfast instead forking at aside to walk down towards the common after breakfast stead of working a decided to walk down towards the common after breakfast stead working a decided to walk down towards the common after breakfast instead working a sudden to walk down towards the common after breakfast instead of working i decided to walk down towards thecommon after breakfast instead of working a decided to walk down towards the common after breakfast instead of working a descended to walk down towards the common after breakfast instead of working at a sudden to walk down towards the common after breakfast instead of working a decided walk down towards the common after breakfast instead of working a decided to walk down towards the common

[0076] This prompt (“Direct top 10 beams”) results in the text shown in the Table below, where LISA corrects three of the four errors present in this utterance.

[0077] In accordance with some embodiments, because CTC loss (e.g., as determined in FIG. 1) may be loosely predictive of WER, overfitting may be used to help the approximate beam search algorithms further increase in reliability. The overfitting of CTC also invites research that explores new sequence-to-sequence loss functions that generalize without mode collapse.

[0078] In accordance with multiple embodiments, inclusion of at least one text modality during training may further boost performance. This may require alignment approaches including but not limited to phoneme duration predictions, similar to the duration / pitch predictor in NaturalSpeech 2. Additional context in the form of longer utterances (e.g. 30s) and / or text transcripts of previous utterances may further allow for LISA to improve the imputation of missing phonetic content.

[0079] Scoring adjustment configurations implemented in accordance some embodiments of the invention are not limited to use within LLMs. Accordingly, it should be appreciated that applications described herein may also be implemented outside the context of speech synthesis.D. Systems for Silent Speech Recognition1. Speech Recognition System

[0080] An example of a Speech Recognition System that evaluates instances of silent speech in accordance with an embodiment of the invention is illustrated in FIG. 4. The speech recognition system 400 includes a communications network 460. The communications network 460 is a network such as the Internet that allows devices connected to the network 460 to communicate with other connected devices. Server systems 410, 440, and 470 are connected to the network 460. Each of the server systems 410, 440, and 470 is a group of one or more servers communicatively connected to one another via internal networks that execute processes that provide cloud services to users over the network 460. One skilled in the art will recognize that a system may exclude certain components and / or include other components that are omitted for brevity without departing from this invention.

[0081] For purposes of this discussion, cloud services are one or more applications that are executed by one or more server systems to provide data and / or executable applications to devices over a network. The server systems 410, 440, and 470 are shown each having three servers in the internal network. However, the server systems 410, 440 and 470 may include any number of servers and any additional number of server systems may be connected to the network 460 to provide cloud services. In accordance with various embodiments of this invention, a speech recognition system that uses systems and methods that recognize silent speech in accordance with an embodiment of the invention may be provided by a process being executed on a single server system and / or a group of server systems communicating over network 460.

[0082] Users may use personal devices 480 and / or mobile devices 420 that connect to the network 460 to perform processes that evaluate speech and / or attempts at speech in accordance with various embodiments of the invention. In the shown embodiment, the personal devices 480 are shown as desktop computers that are connected via a conventional “wired” connection to the network 460. However, the personal device 480 may be a desktop computer, a laptop computer, a smart television, an entertainment gaming console, or any other device that connects to the network 460 via a “wired”connection. The mobile device 420 connects to network 460 using a wireless connection. A wireless connection is a connection that uses Radio Frequency (RF) signals, Infrared signals, or any other form of wireless signaling to connect to the network 460. In the example of this figure, the mobile device 420 is a mobile telephone. However, mobile device 420 may be a mobile phone, Personal Digital Assistant (PDA), a tablet, a smartphone, or any other type of device that connects to network 460 via wireless connection without departing from this invention.

[0083] As can readily be appreciated the specific computing system used for speech recognition and / or decoding is largely dependent upon the requirements of a given application and should not be considered as limited to any specific computing system(s) implementation.2. Learning Elements

[0084] An example of a learning-directed element that executes instructions to perform processes for training models (including but not limited to multimodal and / or large language models) in accordance with certain embodiments of the invention is illustrated in FIG. 5. Learning elements can include (but are not limited to) one or more of mobile devices, cameras, and / or computers. Learning elements 500 may include but are not limited to at least one processor 505, peripherals 510, a network interface 515, and memory 520. One skilled in the art will recognize that learning elements 500 may exclude certain components and / or include other components that are omitted for brevity without departing from this invention.

[0085] The (at least one) processor 505 can include (but is not limited to) a processor, microprocessor, controller, or a combination of processors, microprocessor, and / or controllers that perform instructions stored in the memory 520 to manipulate data stored in the memory. Processor instructions can configure the processor 505 to perform processes in accordance with certain embodiments of the invention. In various embodiments, processor instructions can be stored on a non-transitory computer- readable and / or machine-readable medium.

[0086] Peripherals 510 can include any of a variety of components for capturing speech-related data used in training models, such as (but not limited to) cameras,displays, brain sensors, microphones, and / or other sensors. In a variety of embodiments, peripherals can be used to gather inputs and / or provide outputs of various users that can be stored as sensor data 535. Learning elements 500 can utilize network interfaces 515 to transmit and receive data over a network based upon the instructions performed by processor(s) 505. Peripherals and / or network interfaces in accordance with many embodiments of the invention can be used to gather inputs that can be used to operate systems and / or processes including but not limited to MONA LISA.

[0087] Memory 520 may include but is not limited to a training application 525, model data 530, and sensor data 535. Sensor data 535 in accordance with a variety of embodiments of the invention can include depictions of speech (including but not limited to silent speech attempts) that can be used in evaluation and / or training processes. In certain embodiments, sensor data 535 can include (but is not limited to) video, images, audio, brain scans, etc. In several embodiments, model data 530 can store various parameters and / or weights for various models that can be used for various processes as described in this specification. Model data 530, in accordance with many embodiments of the invention, can be updated through training performed by the training application 525. Training may be performed based on sensor data 535 captured and / or updated (e.g., remotely) in the learning element 500. The training application 525 can (additionally or alternatively) be used to perform methods including but not limited to sensor data 535 analytics and model training.

[0088] Although a specific example of learning element 500 is illustrated in FIG. 5, any of a variety of elements can be utilized to train multimodal models similar to those described herein as appropriate to the requirements of specific applications. Additionally or alternatively, although specific methods of decoding speech are discussed above with respect to EMG signals, other signal recording modalities can be incorporated such as (but not limited to) intracortical neural signals, EEG signals, electrocorticography (ECoG) signals, and / or any other signals as appropriate to the requirements of specific applications of embodiments of the invention.Exemplary Embodiments

[0089] A first embodiment comprising a method for decoding attempted speech. The method obtains a plurality of recordings corresponding to at least one vocalization,wherein each recording of the plurality of recordings comprises a sensor measurement in a unique data modality that is obtained from a speaker during the at least one vocalization. The method encodes the plurality of recordings, using at least one encoder, into a shared latent space, wherein: each of the plurality of recordings comprises a plurality of down- sampled timesteps; and each down-sampled timestep of the plurality of down-sampled timesteps corresponds to a distinct latent embedding. The method aligns the shared latent space, wherein, when the plurality of recordings is obtained concurrently: aligning the shared latent space comprises temporally aligning the distinct latent embeddings across the plurality of encoded recordings; and temporally aligning the distinct latent embeddings across the plurality of encoded recordings is performed using a crosscontrastive loss function. The method transduces the aligned latent space into synthesized speech.

[0090] A second embodiment, including the features of the first embodiment and further comprising that, when temporally aligning the distinct latent embeddings across the plurality of encoded recordings using the cross-contrastive loss function, similarity for the distinct latent embeddings is maximized for similar timesteps and minimized for different timesteps.

[0091] A third embodiment, including the features of the first or the second embodiment and further comprising that when the plurality of recordings is obtained asynchronously: aligning the shared latent space comprises phonemically aligning the distinct latent embeddings for the plurality of encoded recordings; and phonemically aligning the distinct latent embeddings for the plurality of encoded recordings is performed using a supervised temporal contrastive loss function.

[0092] A fourth embodiment, including the features of the third embodiment and further comprising that, when phonemically aligning the distinct latent embeddings across the plurality of encoded recordings using the cross-contrastive loss function, similarity for the distinct latent embedding is maximized for similar phoneme class labels and minimized for different phoneme class labels.

[0093] A fifth embodiment, including the features of the fourth embodiment and further comprising that, the phoneme class labels distinguish between phonemes and silence.

[0094] A sixth embodiment, including the features of any of the third to fifth embodiments and further comprising that phonemically aligning the distinct latent embeddings for the plurality of encoded recordings is performed, at least in part, using a softmax function.

[0095] A seventh embodiment, including the features of the second embodiment, fourth embodiment, or fifth embodiment, and further comprising that the similarity for the distinct latent embedding is determined, at least in part, using dynamic time warping.

[0096] An eighth embodiment, including the features of any of the first to seventh embodiments and further comprising that each of the plurality of down-sampled timesteps represents a temporal offset between: a first recording of the plurality of recordings; and a second recording of the plurality of recordings.

[0097] A ninth embodiment, including the features of the eighth embodiment and further comprising that the first recording of the plurality of recordings comprises an Electromyography (EMG) measurement; and the second recording of the plurality of recordings comprises an audio measurement.

[0098] A tenth embodiment, including the features of any of the first to ninth embodiments and further comprising that the method trains a decoder to decode silent speech using the aligned latent space.

[0099] An eleventh embodiment, including the features of the tenth embodiment and further comprising that an audio-only dataset is used to train at least one of: the at least one encoder; or the decoder.

[0100] A twelfth embodiment, including the features of any of the first to eleventh embodiments and further comprising that the at least one encoder comprises at least one of an artificial neural network (ANN), and a recurrent neural network (RNN), or a specialized neural network.

[0101] A thirteenth embodiment, including the features of any of the first to twelfth embodiments and further comprising that the method trains the at least one encoder using at least one of: Connectionist Temporal Classification (CTC) loss values generated based on aligning the shared latent space; or the aligned shared latent space.

[0102] A fourteenth embodiment, including the features of any of the first to thirteenth embodiments and further comprising that the method improves recognitionaccuracy of the synthesized speech using at least one of: a heuristic search algorithm; or a large language model.

[0103] A fifteenth embodiment, including the features of any of the first to fourteenth embodiments and further comprising that the unique data modality of at least one of the plurality of recordings is selected from the group consisting of brain implant measurements, lip reading measurements, ultrasound measurements, magnetic resonance imaging (MRI) measurements, Functional Near-Infrared Spectroscopy (fNIRS) measurements, magnetoencephalography (MEG) measurements, electroencephalogram (EEG) measurements, radar measurements, strain sensor measurements, and non-audible murmur measurements.

[0104] A sixteenth embodiment comprising a non-transitory computer-readable medium comprising instructions that, when executed, are configured to cause a processor to perform a process for decoding attempted speech. The processor obtains a plurality of recordings corresponding to at least one vocalization, wherein each recording of the plurality of recordings comprises a sensor measurement in a unique data modality that is obtained from a speaker during the at least one vocalization. The processor encodes the plurality of recordings, using at least one encoder, into a shared latent space, wherein: each of the plurality of recordings comprises a plurality of down-sampled timesteps; and each down-sampled timestep of the plurality of down-sampled timesteps corresponds to a distinct latent embedding. The processor aligns the shared latent space, wherein, when the plurality of recordings is obtained concurrently: aligning the shared latent space comprises temporally aligning the distinct latent embeddings across the plurality of encoded recordings; and temporally aligning the distinct latent embeddings across the plurality of encoded recordings is performed using a cross-contrastive loss function. The processor transduces the aligned latent space into synthesized speech.

[0105] A seventeenth embodiment, including the features of the sixteenth embodiment and further comprising that, when temporally aligning the distinct latent embeddings across the plurality of encoded recordings using the cross-contrastive loss function, similarity for the distinct latent embeddings is maximized for similar timesteps and minimized for different timesteps.

[0106] An eighteenth embodiment, including the features of the sixteenth or the seventeenth embodiment and further comprising that when the plurality of recordings is obtained asynchronously: aligning the shared latent space comprises phonemically aligning the distinct latent embeddings for the plurality of encoded recordings; and phonemically aligning the distinct latent embeddings for the plurality of encoded recordings is performed using a supervised temporal contrastive loss function.

[0107] A nineteenth embodiment, including the features of the eighteenth embodiment and further comprising that, when phonemically aligning the distinct latent embeddings across the plurality of encoded recordings using the cross-contrastive loss function, similarity for the distinct latent embedding is maximized for similar phoneme class labels and minimized for different phoneme class labels.

[0108] A twentieth embodiment, including the features of the nineteenth embodiment and further comprising that, the phoneme class labels distinguish between phonemes and silence.

[0109] A twenty-first embodiment, including the features of any of the eighteenth to twentieth embodiments and further comprising that phonemically aligning the distinct latent embeddings for the plurality of encoded recordings is performed, at least in part, using a softmax function.

[0110] A twenty-second embodiment, including the features of the seventeenth embodiment, nineteenth embodiment, or twentieth embodiment, and further comprising that the similarity for the distinct latent embedding is determined, at least in part, using dynamic time warping.

[0111] A twenty-third embodiment, including the features of any of the sixteenth to twenty-second embodiments and further comprising that each of the plurality of down- sampled timesteps represents a temporal offset between: a first recording of the plurality of recordings; and a second recording of the plurality of recordings.

[0112] A twenty-fourth embodiment, including the features of the twenty-third embodiment and further comprising that the first recording of the plurality of recordings comprises an Electromyography (EMG) measurement; and the second recording of the plurality of recordings comprises an audio measurement.

[0113] A twenty -fifth embodiment, including the features of any of the sixteenth to twenty-fourth embodiments and further comprising that the processor trains a decoder to decode silent speech using the aligned latent space.

[0114] A twenty-sixth embodiment, including the features of the twenty-fifth embodiment and further comprising that an audio-only dataset is used to train at least one of: the at least one encoder; or the decoder.

[0115] A twenty-seventh embodiment, including the features of any of the sixteenth to twenty-sixth embodiments and further comprising that the at least one encoder comprises at least one of an artificial neural network (ANN), and a recurrent neural network (RNN), or a specialized neural network.

[0116] A twenty-eighth embodiment, including the features of any of the sixteenth to twenty-seventh embodiments and further comprising that the processor trains the at least one encoder using at least one of: Connectionist Temporal Classification (CTC) loss values generated based on aligning the shared latent space; or the aligned shared latent space.

[0117] A twenty-ninth embodiment, including the features of any of the sixteenth to twenty-eighth embodiments and further comprising that the processor improves recognition accuracy of the synthesized speech using at least one of: a heuristic search algorithm; or a large language model.

[0118] A thirtieth embodiment, including the features of any of the sixteenth to twenty-ninth embodiments and further comprising that the unique data modality of at least one of the plurality of recordings is selected from the group consisting of brain implant measurements, lip reading measurements, ultrasound measurements, magnetic resonance imaging (MRI) measurements, Functional Near-Infrared Spectroscopy (fNIRS) measurements, magnetoencephalography (MEG) measurements, electroencephalogram (EEG) measurements, radar measurements, strain sensor measurements, and non-audible murmur measurements.

[0119] A thirtieth embodiment comprising a system for decoding attempted speech, comprising: at least one sensor; a network interface; at least one encoder; and a processor configured to execute instructions. The processor obtains a plurality of recordings corresponding to at least one vocalization, wherein each recording of theplurality of recordings comprises a sensor measurement, measured by one or more sensors of the at least one sensor, in a unique data modality that is obtained from a speaker during the at least one vocalization. The processor encodes the plurality of recordings, using at least one encoder, into a shared latent space, wherein: each of the plurality of recordings comprises a plurality of down-sampled timesteps; and each down- sampled timestep of the plurality of down-sampled timesteps corresponds to a distinct latent embedding. The processor aligns the shared latent space, wherein, when the plurality of recordings is obtained concurrently: aligning the shared latent space comprises temporally aligning the distinct latent embeddings across the plurality of encoded recordings; and temporally aligning the distinct latent embeddings across the plurality of encoded recordings is performed using a cross-contrastive loss function. The processor transduces the aligned latent space into synthesized speech.

[0120] A thirty-second embodiment, including the features of the thirty-first embodiment and further comprising that, when temporally aligning the distinct latent embeddings across the plurality of encoded recordings using the cross-contrastive loss function, similarity for the distinct latent embeddings is maximized for similar timesteps and minimized for different timesteps.

[0121] A thirty-third embodiment, including the features of the thirty-first or the thirty-second embodiment and further comprising that when the plurality of recordings is obtained asynchronously: aligning the shared latent space comprises phonemically aligning the distinct latent embeddings for the plurality of encoded recordings; and phonemically aligning the distinct latent embeddings for the plurality of encoded recordings is performed using a supervised temporal contrastive loss function.

[0122] A thirty-fourth embodiment, including the features of the thirty-third embodiment and further comprising that, when phonemically aligning the distinct latent embeddings across the plurality of encoded recordings using the cross-contrastive loss function, similarity for the distinct latent embedding is maximized for similar phoneme class labels and minimized for different phoneme class labels.

[0123] A thirty-fifth embodiment, including the features of the thirty-fourth embodiment and further comprising that, the phoneme class labels distinguish between phonemes and silence.

[0124] A thirty-sixth embodiment, including the features of any of the thirty-third to thirty-fifth embodiments and further comprising that phonemically aligning the distinct latent embeddings for the plurality of encoded recordings is performed, at least in part, using a softmax function.

[0125] A thirty-seventh embodiment, including the features of the thirty-second embodiment, thirty-fourth embodiment, or thirty-fifth embodiment, and further comprising that the similarity for the distinct latent embedding is determined, at least in part, using dynamic time warping.

[0126] A thirty-eighth embodiment, including the features of any of the thirty-first to thirty-seventh embodiments and further comprising that each of the plurality of down- sampled timesteps represents a temporal offset between: a first recording of the plurality of recordings; and a second recording of the plurality of recordings.

[0127] A thirty-ninth embodiment, including the features of the thirty-eighth embodiment and further comprising that the first recording of the plurality of recordings comprises an Electromyography (EMG) measurement; and the second recording of the plurality of recordings comprises an audio measurement.

[0128] A fortieth embodiment, including the features of any of the thirty-first to thirty-ninth embodiments and further comprising that the processor trains a decoder to decode silent speech using the aligned latent space.

[0129] A forty-first embodiment, including the features of the fortieth embodiment and further comprising that an audio-only dataset is used to train at least one of: the at least one encoder; or the decoder.

[0130] A forty-second embodiment, including the features of any of the thirty-first to forty-first embodiments and further comprising that the at least one encoder comprises at least one of an artificial neural network (ANN), and a recurrent neural network (RNN), or a specialized neural network.

[0131] A forty-third embodiment, including the features of any of the thirty-first to forty-second embodiments and further comprising that the processor trains the at least one encoder using at least one of: Connectionist Temporal Classification (CTC) loss values generated based on aligning the shared latent space; or the aligned shared latent space.

[0132] A forty-fourth embodiment, including the features of any of the thirty-first to forty-third embodiments and further comprising that the processor improves recognition accuracy of the synthesized speech using at least one of: a heuristic search algorithm; or a large language model.

[0133] A forty-fifth embodiment, including the features of any of the thirty-first to forty-fourth embodiments and further comprising that the unique data modality of at least one of the plurality of recordings is selected from the group consisting of brain implant measurements, lip reading measurements, ultrasound measurements, magnetic resonance imaging (MRI) measurements, Functional Near-Infrared Spectroscopy (fNIRS) measurements, magnetoencephalography (MEG) measurements, electroencephalogram (EEG) measurements, radar measurements, strain sensor measurements, and non-audible murmur measurements.

[0134] While specific systems and methods of silent speech interfacing are discussed above, many different methods of decoding silent speech can be implemented in accordance with many different embodiments of the invention. It is therefore to be understood that the present invention may be practiced in ways other than specifically described, without departing from the scope and spirit of the present invention. Thus, embodiments of the present invention should be considered in all respects as illustrative and not restrictive. Accordingly, the scope of the invention should be determined not by the embodiments illustrated, but by the appended claims and their equivalents.

Claims

WHAT IS CLAIMED IS:1 . A method for decoding attempted speech, comprising: obtaining a plurality of recordings corresponding to at least one vocalization, wherein each recording of the plurality of recordings comprises a sensor measurement in a unique data modality that is obtained from a speaker during the at least one vocalization; encoding the plurality of recordings, using at least one encoder, into a shared latent space, wherein: each of the plurality of recordings comprises a plurality of down-sampled timesteps; and each down-sampled timestep of the plurality of down-sampled timesteps corresponds to a distinct latent embedding; aligning the shared latent space, wherein, when the plurality of recordings is obtained concurrently: aligning the shared latent space comprises temporally aligning the distinct latent embeddings across the plurality of encoded recordings; and temporally aligning the distinct latent embeddings across the plurality of encoded recordings is performed using a cross-contrastive loss function; and transducing the aligned latent space into synthesized speech.

2. The method of claim 1 , wherein, when temporally aligning the distinct latent embeddings across the plurality of encoded recordings using the cross-contrastive loss function, similarity for the distinct latent embeddings is maximized for similar timesteps and minimized for different timesteps.

3. The method of claim 1 or claim 2, wherein, when the plurality of recordings is obtained asynchronously: aligning the shared latent space comprises phonemically aligning the distinct latent embeddings for the plurality of encoded recordings; and phonemically aligning the distinct latent embeddings for the plurality of encoded recordings is performed using a supervised temporal contrastive loss function.

4. The method of claim 3, wherein, when phonemically aligning the distinct latent embeddings across the plurality of encoded recordings using the cross-contrastive loss function, similarity for the distinct latent embedding is maximized for similar phoneme class labels and minimized for different phoneme class labels.

5. The method of claim 4, wherein the phoneme class labels distinguish between phonemes and silence.

6. The method of any of claims 3-5, wherein phonemically aligning the distinct latent embeddings for the plurality of encoded recordings is performed, at least in part, using a softmax function.

7. The method of any of claims 2, 4, or 5, wherein the similarity for the distinct latent embedding is determined, at least in part, using dynamic time warping.

8. The method of any of claims 1-7, wherein each of the plurality of down-sampled timesteps represents a temporal offset between: a first recording of the plurality of recordings; and a second recording of the plurality of recordings.

9. The method of claim 8, wherein: the first recording of the plurality of recordings comprises an Electromyography (EMG) measurement; and the second recording of the plurality of recordings comprises an audio measurement.

10. The method of any of claims 1 -9, further comprising training a decoder to decode silent speech using the aligned latent space.

11. The method of claim 10, wherein an audio-only dataset is used to train at least one of: the at least one encoder; or the decoder.

12. The method of any of claims 1-11 , wherein the at least one encoder comprises at least one of an artificial neural network (ANN), and a recurrent neural network (RNN), or a specialized neural network.

13. The method of any of claims 1-12, further comprising, training the at least one encoder using at least one of:Connectionist Temporal Classification (CTC) loss values generated based on aligning the shared latent space; or the aligned shared latent space.

14. The method of any of claims 1 -13, further comprising improving recognition accuracy of the synthesized speech using at least one of: a heuristic search algorithm; or a large language model.

15. The method of any of claims 1 -1 , wherein the unique data modality of at least one of the plurality of recordings is selected from the group consisting of brain implant measurements, lip reading measurements, ultrasound measurements, magnetic resonance imaging (MRI) measurements, Functional Near-Infrared Spectroscopy (fNIRS) measurements, magnetoencephalography (MEG) measurements, electroencephalogram (EEG) measurements, radar measurements, strain sensor measurements, and non-audible murmur measurements.

16. A non-transitory computer-readable medium comprising instructions that, when executed, are configured to cause a processor to perform a process for decoding attempted speech, comprising: obtaining a plurality of recordings corresponding to at least one vocalization, wherein each recording of the plurality of recordings comprises a sensor measurement in a unique data modality that is obtained from a speaker during the at least one vocalization; encoding the plurality of recordings, using at least one encoder, into a shared latent space, wherein: each of the plurality of recordings comprises a plurality of down-sampled timesteps; and each down-sampled timestep of the plurality of down-sampled timesteps corresponds to a distinct latent embedding; aligning the shared latent space, wherein, when the plurality of recordings is obtained concurrently: aligning the shared latent space comprises temporally aligning the distinct latent embeddings across the plurality of encoded recordings; and temporally aligning the distinct latent embeddings across the plurality of encoded recordings is performed using a cross-contrastive loss function; and transducing the aligned latent space into synthesized speech.

17. The non-transitory computer-readable medium of claim 16, wherein, when temporally aligning the distinct latent embeddings across the plurality of encoded recordings using the cross-contrastive loss function, similarity for the distinct latent embeddings is maximized for similar timesteps and minimized for different timesteps.

18. The non-transitory computer-readable medium of claim 16 or claim 17, wherein, when the plurality of recordings is obtained asynchronously: aligning the shared latent space comprises phonemically aligning the distinct latent embeddings for the plurality of encoded recordings; and phonemically aligning the distinct latent embeddings for the plurality of encoded recordings is performed using a supervised temporal contrastive loss function.

19. The non-transitory computer-readable medium of claim 18, wherein, when phonemically aligning the distinct latent embeddings across the plurality of encoded recordings using the cross-contrastive loss function, similarity for the distinct latent embedding is maximized for similar phoneme class labels and minimized for different phoneme class labels.

20. The non-transitory computer-readable medium of claim 19, wherein the phoneme class labels distinguish between phonemes and silence.

21. The non-transitory computer-readable medium of any of claims 18-20, wherein phonemically aligning the distinct latent embeddings for the plurality of encoded recordings is performed, at least in part, using a softmax function.

22. The non-transitory computer-readable medium of any of claims 17, 19, or 20, wherein the similarity for the distinct latent embedding is determined, at least in part, using dynamic time warping.

23. The non-transitory computer-readable medium of any of claims 16-22, wherein each of the plurality of down-sampled timesteps represents a temporal offset between: a first recording of the plurality of recordings; and a second recording of the plurality of recordings.

24. The non-transitory computer-readable medium of claim 23, wherein: the first recording of the plurality of recordings comprises an Electromyography (EMG) measurement; and the second recording of the plurality of recordings comprises an audio measurement.

25. The non-transitory computer-readable medium of any of claims 16-24, further comprising training a decoder to decode silent speech using the aligned latent space.

26. The non-transitory computer-readable medium of claim 25, wherein an audio-only dataset is used to train at least one of: the at least one encoder; or the decoder.

27. The non-transitory computer-readable medium of any of claims 16-26, wherein the at least one encoder comprises at least one of an artificial neural network (ANN), and a recurrent neural network (RNN), or a specialized neural network.

28. The non-transitory computer-readable medium of any of claims 16-27, further comprising, training the at least one encoder using at least one of:Connectionist Temporal Classification (CTC) loss values generated based on aligning the shared latent space; or the aligned shared latent space.

29. The non-transitory computer-readable medium of any of claims 16-28, further comprising improving recognition accuracy of the synthesized speech using at least one of: a heuristic search algorithm; or a large language model.

30. The non-transitory computer-readable medium of any of claims 16-29, wherein the unique data modality of at least one of the plurality of recordings is selected from the group consisting of brain implant measurements, lip reading measurements, ultrasound measurements, magnetic resonance imaging (MRI) measurements, Functional NearInfrared Spectroscopy (fNIRS) measurements, magnetoencephalography (MEG) measurements, electroencephalogram (EEG) measurements, radar measurements, strain sensor measurements, and non-audible murmur measurements.

31. A system for decoding attempted speech, comprising: at least one sensor; a network interface; at least one encoder; and a processor configured to execute instructions comprising: obtaining a plurality of recordings corresponding to at least one vocalization, wherein each recording of the plurality of recordings comprises a sensor measurement, measured by one or more sensors of the at least one sensor, in a unique data modality that is obtained from a speaker during the at least one vocalization; encoding the plurality of recordings, using at least one encoder, into a shared latent space, wherein: each of the plurality of recordings comprises a plurality of down- sampled timesteps; and each down-sampled timestep of the plurality of down-sampled timesteps corresponds to a distinct latent embedding; aligning the shared latent space, wherein, when the plurality of recordings is obtained concurrently: aligning the shared latent space comprises temporally aligning the distinct latent embeddings across the plurality of encoded recordings; and temporally aligning the distinct latent embeddings across the plurality of encoded recordings is performed using a cross-contrastive loss function; and transducing the aligned latent space into synthesized speech.

32. The system of claim 31 , wherein, when temporally aligning the distinct latent embeddings across the plurality of encoded recordings using the cross-contrastive loss function, similarity for the distinct latent embeddings is maximized for similar timesteps and minimized for different timesteps.

33. The system of claim 31 or claim 32, wherein, when the plurality of recordings is obtained asynchronously: aligning the shared latent space comprises phonemically aligning the distinct latent embeddings for the plurality of encoded recordings; and phonemically aligning the distinct latent embeddings for the plurality of encoded recordings is performed using a supervised temporal contrastive loss function.

34. The system of claim 33, wherein, when phonemically aligning the distinct latent embeddings across the plurality of encoded recordings using the cross-contrastive loss function, similarity for the distinct latent embedding is maximized for similar phoneme class labels and minimized for different phoneme class labels.

35. The system of claim 34, wherein the phoneme class labels distinguish between phonemes and silence.

36. The system of any of claims 33-35, wherein phonemically aligning the distinct latent embeddings for the plurality of encoded recordings is performed, at least in part, using a softmax function.

37. The system of any of claims 32, 34, or 35, wherein the similarity for the distinct latent embedding is determined, at least in part, using dynamic time warping.

38. The system of any of claims 31 -37, wherein each of the plurality of down-sampled timesteps represents a temporal offset between: a first recording of the plurality of recordings; and a second recording of the plurality of recordings.

39. The system of claim 38, wherein: the first recording of the plurality of recordings comprises an Electromyography (EMG) measurement; and the second recording of the plurality of recordings comprises an audio measurement.

40. The system of any of claims 31 -39, further comprising training a decoder to decode silent speech using the aligned latent space.

41. The system of claim 40, wherein an audio-only dataset is used to train at least one of: the at least one encoder; or the decoder.

42. The system of any of claims 31 -41 , wherein the at least one encoder comprises at least one of an artificial neural network (ANN), and a recurrent neural network (RNN), or a specialized neural network.

43. The system of any of claims 31 -42, further comprising, training the at least one encoder using at least one of:Connectionist Temporal Classification (CTC) loss values generated based on aligning the shared latent space; or the aligned shared latent space.

44. The system of any of claims 31 -43, further comprising improving recognition accuracy of the synthesized speech using at least one of: a heuristic search algorithm; or a large language model.

45. The system of any of claims 31 -44, wherein the unique data modality of at least one of the plurality of recordings is selected from the group consisting of brain implant measurements, lip reading measurements, ultrasound measurements, magnetic resonance imaging (MRI) measurements, Functional Near-Infrared Spectroscopy (fNIRS) measurements, magnetoencephalography (MEG) measurements, electroencephalogram (EEG) measurements, radar measurements, strain sensor measurements, and non-audible murmur measurements.

Images