Systems and methods involving aspects of neural interfaces, open-vocabulary, continuous imagined speech processing, ai models for human-ai interaction and / or other features

Abstract

Systems and methods are provided for continuous, open-vocabulary decoding of imagined speech from non-invasive brain signals. Consistent with aspects of the disclosed technology, a user may silently compose sentences on diverse topics while high-density fNIRS, MEG, fMRI or similar sensors record neural activity. In one illustrative implementation, the recorded time-series are passed through a sequence-to-sequence transformer (or other encoder or processor) that projects the signals into the embedding space of a large language model. Such brain-derived embeddings may be combined with selectable textual context and injected as a prompt into a LLM, which may complete the sentence, yielding continuous text that reflects the user's internal speech. According to additional aspects, alignment across multiple participants plus rapid per-user fine-tuning improves accuracy and / or generalizability. In our example implementation, experimental results show statistically significant gains in BLEU and BERT similarity metrics. Further, the disclosed technology delivers a portable, adaptable path to non- invasive, thought-driven interaction with Al assistants, wearable controllers and other computing devices.

Description

SYSTEMS AND METHODS INVOLVING ASPECTS OF NEURAL INTERFACES, OPEN- VOCABULARY, CONTINUOUS IMAGINED SPEECH PROCESSING, Al MODELS FOR HUMAN-AI INTERACTION AND / OR OTHER FEATURESCross-Reference to Related and Appendix Information

[0001] This application claims benefit / priority under the provisions of the Patent Cooperation Treaty, and all other applicable National Stage provisions, of U.S. Provisional Application No. 63 / 663,056, filed June 21, 2024, the contents of which are incorporated herein by reference and inclusion in their entirety.

[0002] This application also includes Appendix materials enclosed herewith to even more clearly disclose shading and / or coloration associated with tables from the Specification.BackgroundField

[0003] The disclosed technology relates, inter alia, to the field of human- Al interaction(s) via implementation of systems and methods involving detection, processing, and / or decoding of brain data.Description of Related Technology

[0004] In the coming decade, artificial intelligence systems are improving exponentially and are set to revolutionize every industry and facet of human life. Building communication systems that enable seamless and symbiotic communication between humans and Al agents is increasingly important.

[0005] With regard to drawbacks of existing technology, at present humans must interact with computers via typing, mouse click and voice. This is also true for interacting with Al assistants. Existing solutions also sometime use neural interface systems that, e.g., are surgical interfaces, which decode motor movements or can enable mouse clicks or single word or phrase decoding. However, the most fluid form of human computer and human- Al interaction, e.g., being able to directly relay thoughts in our mind to the Al, would be ideal. This would facilitate superior human-AI symbiosis and would enable human goals and ideas to be rapidly and effectively transferred to Al and computers, dramatically improving efficiency, productivity and enjoyment. It would have a profound impact in consumer electronics, medical domains and in all other industries.

[0006] Various aspects of the innovations set forth herein overcome these and other drawbacks and otherwise advance the field of human- Al interaction, inter alia, by disclosing proprietary Al models and neural interface systems, methods and algorithms, generically referred to as MindSpeech. Among other things, the disclosed technology enables open-vocabulary, continuous decoding for imagined speech.Overview of Illustrative Aspects

[0007] As set forth in detail, below, systems and methods of the present technology enable openvocabulary, continuous imagined speech and / or otherwise disclose features, functionality and innovations related thereto. Among other things, the disclosed technology allows a user to freely imagine language in their mind and for it to be outputted as text continuously. According to certain embodiments, various implementations herein may utilize brain data from portable headsets, which could be used in a consumer product system, i.e., no surgery is required.Consistent with such embodiments and implementations, the disclosed technology may utilize high-density functional near-infrared spectroscopy (fNIRS) data as well as development and use of Al models capable of decoding imagined speech. Further, the disclosed technology may be configured for implemented and use with other types of brain data.

[0008] The disclosed technology and / or related approaches herein also include and / or involve development and / or use of Al models that are novel and particularly innovative as leveraged with and / or applied to brain data in the implementations herein for imagined speech. Additional innovative aspects include and / or involve creation and / or implementation of novel word cloud paradigms for data collection, which, inter alia, improve the quality and variety of imagined sentences generated by participants and / or cover a broad semantic space, among other advantages.

[0009] Further, the disclosed technology enables combining high-density fNIRS with advanced Al techniques to develop non-invasive, accurate BCIs for imagined speech, which, inter alia, significantly achieves or established a basis for advanced human-machine interaction.Brief Description of the Drawings

[0010] Various embodiments of the present disclosure may be further explained with reference to the attached drawings, wherein like structures are referred to by like numerals throughout the several views. The drawings shown are not necessarily to scale, with emphasis instead generally being placed upon illustrating the principles of the present disclosure. Therefore, specific structural and functional details disclosed herein are not to be interpreted as limiting, but merelyas a representative basis for teaching one skilled in the art to variously employ one or more illustrative embodiments.

[0011] FIGs. 1 A-1B depict a representation illustrating aspects associated with systems and methods herein implementing an exemplary word cloud paradigm, consistent with various exemplary aspects of one or more implementations of the disclosed technology.

[0012] FIG. 2 depicts one illustrative whole head arrangement of an exemplary high-density fNIRS montage, consistent with various exemplary aspects of one or more implementations of the disclosed technology.

[0013] FIGs. 3A-3C depict representative block / sequence diagrams illustrating an exemplary prompt tuning process in continuous imagined speech decoding, consistent with various exemplary aspects of one or more implementations of the disclosed technology.

[0014] FIG. 4 depicts illustrative FIR model coefficient values across delays showing estimated shapes of responses for participants across word cloud and resting conditions, consistent with various exemplary aspects of one or more implementations of the disclosed technology.

[0015] FIG. 5 depicts an illustrative summary of representative classifier model performance in classifying imagined speech versus rest for certain exemplary brain / fNIRS data, consistent with various exemplary aspects of one or more implementations of the disclosed technology.

[0016] FIG. 6 depicts illustrative color imaging of surface cortex plots showing HbO activations with the contrast word cloud vis-a-vis rest across all participants, consistent with various exemplary aspects of one or more implementations of the disclosed technology. The colorbar indicates z-score values.Detailed Description of Illustrative Implementations

[0017] Systems and methods involving computer neural interfaces are disclosed, such as some enabling users of decoding imaged sentences through a non-invasive neural interface.Implementation involving custom brain encoding models and brain signal guided inputs demonstrated improved language and semantic similarity in generated text continuation compared to permutation conditions. Aspects of the disclosed technology allow systems and methods herein to isolate imagined speech from other cognitive functions, such as memory, eliminate the reliance on memorization of preset sentences, and / or minimize the need for training to collect imagined speech data, among other improvements and benefits. Further, the present systems and methods may be configured to leverage brain-computer interface and / or non- invasive wearable device aspects to provide enhanced user interactions for next-generationwearable devices, controllers, and / or other computing components based on the human thoughts, brain signals, and / or mind activity that are detected and processed.

[0018] Systems and methods are provided for continuous, open-vocabulary decoding of imagined speech from non-invasive brain signals. Using a “word-cloud” paradigm, a user silently composes sentences on diverse topics while high-density fNIRS, MEG, fMRI or similar sensors record neural activity. The recorded time-series are passed through a sequence-to- sequence transformer (or other encoder) that projects the signals into the embedding space of a large language model. These brain-derived embeddings are combined with selectable textual context — ranging from no context, to a topic cue, to the first words of the sentence — and injected as a prompt into a LLM. This method enables the LLM to complete the sentence, yielding continuous text that reflects the user’s internal speech. Alignment across multiple participants plus rapid per-user fine-tuning improves accuracy and generalisability. In our example implementation, experimental results with 48 x 48 whole-head fNIRS show statistically significant gains in BLEU and BERT similarity metrics over baselines. The technology delivers a portable, adaptable path to non-invasive, thought-driven interaction with Al assistants, wearable controllers and other computing devices.Novel Data Collection Technologies / Paradigm

[0019] Various aspects of the disclosed technology may include and / or involve novel systems, methodologies and / or techniques for imagined speech data collection (via fNIRS or other modalities set forth herein) that captures a wide variety of semantic meaning. According to embodiments herein, the novel technology may prompt participants to imagine sentences of different topics by providing topic words and keywords during the task. Further, the participants may also be asked to type out the imagined sentences as ground truth. Differently to our work with discrete sentence classification (i.e., MindGPT, disclosed in parallel application(s)), features and functionality of the presently disclosed technology allow systems and methods herein to isolate imagined speech from other cognitive functions, such as memory, etc., e.g., by eliminating the reliance on memorization of preset sentences, and / or minimizing the need for training to collect imagined speech data, among other benefits. Secondly, according to some embodiments, implementations herein may utilize a finite impulse response (FIR) model to systematically identify the appropriate BOLD delay for imagined speech fNIRS data. Finally, the disclosed technology may produce and / or utilizes a prompt tuning-based model for natural text generation directly from imagined speech fNIRS brain signals. With such improved brainencoding model, and / or the associated prompt tuning based approach, brain signal guided inputs may generate sentences higher in metrics with some statistical significance that measure both language and semantic similarity, compared to permutation inputs with randomized context input - brain signals pairs.

[0020] According to the disclosed technology, systems and methods herein may generate and / or perform processing involving a processing sequence (task) specifically configured / programmed for continuous imagined speech generation, referred to herein as a ‘word cloud’ task (see, e.g., an example representation shown in Figures 1 A-1B). Consistent with the disclosed technology, such word cloud task may include participants being presented with a selection of words on the screen. Next, the disclosed technology is implemented to have participants imagine a sentence from the given words sequentially.

[0021] FIGs. 1 A-1B depict representation(s) illustrating aspects associated with systems and methods herein implementing an exemplary word cloud paradigm, consistent with various exemplary aspects of one or more implementations of the disclosed technology. Referring to the example of Figure 1 A, an illustrative participant imagines a sentence given a topic word and a highlighted keyword in the ‘word cloud’ processing / display sequence shown. After the imagine period concludes for the topic, they are prompted to type out the imagined sentences. In such embodiment, the brain signals during imagined speech and the ground truth texts are used for decoder training. Referring to Figure IB, an example topic block is illustrated, as may be shown to the participant during certain implementations. The topic word is the bolded word with larger font in the middle, and, according to this illustrative implementation, the surrounding keywords are selectively highlighted in random order to prompt the participant to imagine a sentence related to both the topic and the keyword. Further, here, the participant may then then asked to type out the sentences imagined in the block at the end. Such features and functionality may be achieved via other techniques and display(s).

[0022] In the example implementation shown in Figures 1 A-1B, a central word may be displayed graphically, such as in a larger font, etc., to indicate that it is the topic word, while other words may be displayed differently (e.g., in a smaller font surrounding the topic word, etc.) to show that they are keywords related to the topic. During use of the disclosed technology, various keywords (such as ‘thriving’ here) may be selected randomly, one at a time, by providing the word in bold font, each for a set period of time, e.g., 7 seconds, or another set time. As the keyword appears in the bold font, the system may prompt / cause participants to imagine asentence relating both the topic word and the keyword. According to various implementations herein, the sentences did not need to be created using all keywords. Consistent with certain exemplary embodiments, participants were required to use only the last bolded keyword; the topic word. After the presentation of all the keywords, the task was completed for the given topic and participants are then instructed to type on the computer using a keyboard the sentences they imagined relating the topic to each keyword. This input data was used as a ground truth for the training of the decoder in the example illustrated. In some example sequences, the typing period was not timed, and participants were prompted to move on to the next topic after they had completed all typing. With such methods and input data, the word cloud task(s) may perform processing that taps into participants’ imagined speech cognitive function and is distinctly different from silent reading, where participants are asked to read provided sentences from a screen. Among other advantages, there is no heavy reliance on memorization according to such word cloud task processing innovations or need for pacing imagined speech words at a specific word / minute rate using a metronome, removing any contamination of other cognitive functions, such as memory or auditory processing, from the disclosed technology to isolate imagined speech, also providing a new way to prompt for imagined speech in participants without extensive prior training often needed / required.Exemplary Confirmatory Experiment / Study

[0023] In one confirming study, a total of 272 distinct topics were included. The selection of topics covered a wide range of concrete and abstract concepts in the form of concise, informative sentences and were used in this study given that the study had established decodability of the fMRI activities from silent reading of these sentences. In addition, 125 topics were generated, such as by using keywords from the NGSL spoken, which covers over 90% of the most frequently used words in the English language. OpenAI's GPT-4 model (2024) was then used to generate sentences of similar lengths, and then keywords are extracted from these sentences. These GPT-4 generated sentences and keywords were generally less formal, reflecting the verbal nature of the NGSL Spoken data, thereby balancing the full dataset in terms of verbal styles, for a more complete representation of the English language.

[0024] In this exemplary confirmatory experiment, a word cloud task of three topics formed a run. Each run contained a resting trial of 20 seconds, which was randomly added at the beginning, middle or end of each run. During resting trials, the word ‘baseline’ was displayed on the screen throughout the trial. A topic break of 10 seconds was utilized if no resting trial wasadded in between topics. On each day of the experiment, participants were asked to complete as many runs as possible in an hour, before having the sensor cap removed for a 20 minute break (or longer if required). All the runs within an experimental day were grouped into a session. All 272 topics were used for the word cloud task in a randomized order. Due to the limited amount of experimental time, participants completed only a portion of the topics (see Table 1 for topic and sentence statistics). Note that, in this exemplary confirmatory experiment, only brain signals obtained during the imagined period (i.e., keyword highlighting) were used in the decoder training, not the typing period. Here, for example, the movement artefacts in the brain signals could thereby be minimized.

[0025] In this experiment, we used a commercially available Continuous-Wave (CW) high- density 48x48 fNIRS system (NIRx Inc.) for collecting neurovascular data. The montage provided full-head coverage, including 48 sources and 47 detectors (the extra detector is used for the short-distance channels). The system consists of a total of 388 channels (194 source wavelength at 760 nm and 194 source wavelength at 850 nm), with channel distances ranging from ~21 mm to ~42 mm, and 8 short-distance channels (channel distances: < 10 mm). A sampling rate of 5.9Hz was used.

[0026] One such example of how systems and methods herein may acquire such data is shown in Figure 2. However, implementations herein may also utilize other types of NIRS setup, EEG setups, as well as fMRI and other neuroimaging modalities, if desired. Brain data collected during the experiment was streamed through NIRx acquisition software, Aurora fNIRs (NIRx Medical Technologies LLC), and saved in XDF format. The saved data was then loaded and subjected to a series of preprocessing techniques. According to the disclosed technology, for example, for each run, data may undergo conversion of raw signals by a variety of techniques, such as conversion to optical density, detrending, short channel regression correction, motion artefact correction, conversion to haemoglobin concentration using a partial pathlength factor (ppf), such as 6 in the study though other factors may be used, bandpass filtering, such as between 0.01 and 0.7 Hz though other ranges may be used. For an imagined sentence recorded in 7 seconds, the corresponding data has a sequence length of 42 time points by 388 channels (194 oxy-haemoglobin and 194 deoxy-haemoglobin channels).

[0027] As an illustrative next step of the exemplary study, a Finite Impulse Response (FIR) model was employed to estimate the brain's hemodynamic response to different stimuli without assuming a specific shape for the response function. Various other models and estimationtechniques may be used according to embodiments herein. Given that the brain’s haemodynamic response may vary across individual participants and experimental tasks, a FIR model was estimated for each participant, here in this example comprising two conditions of word cloud and rest. Further, the raw intensity data was converted to optical density and then to haemoglobin concentrations and resampled to 0.5Hz. In some embodiments, such FIR model may use a design matrix with columns representing different time lags of the stimulus, allowing the response at each time point following the stimulus to be captured. In a first level design matrix, the hemodynamic response function (HRF) was set to "fir" with 5 delays. A cosine drift model with a high-pass filter set at 0.01 Hz was applied to account for low-frequency drifts. The design matrix was created without oversampling. Averaged short channel sequences were used as nuisance regressors in a General Linear Model (GLM) to isolate brain activity from systemic physiology. Individual design matrices were created and estimations were performed for each run, followed by the concatenation of all the runs’ results for each participant. Finally, mixed- effects linear models were used to estimate coefficients separately for each delay, task, and oxy / deoxygenated haemoglobin.

[0028] Although one specific, illustrative implementation is disclosed above for the exemplary study, other delays, matrix designs, response functions, models, modeling techniques, filters, and / or other parameters and different techniques may be utilized in other emobodiments consistent with the disclosed technology.Continuous Imagined Speech Decoding Model

[0029] According to certain embodiments herein, to decode continuous imagined speech from non-invasive fNTRS brain signals, the decoding model needs to be able to a) extract semantic information from the brain signals, and b) generate legible continuous speech given the semantic information.Prompt tuning for foundational LLMs

[0030] Existing tuning-based methodologies have been previously employed to harness the knowledge embedded within large language models (LLMs) for decoding time-series data. However, such methodologies have various drawbacks and these approaches generally encompass the processes of segmenting and tokenizing time series signals and associated textual data, followed by fine-tuning the models for specific tasks.

[0031] In contrast, consistent with the disclosed technology, prompt tuning processing may be performed and may comprise steps such as the following, such as from the exemplary study:Collecting brain recordings during the imagined speech condition, during which participants silently imagined a sentence based on a given topic word and a keyword (see, e.g., ‘Word Cloud’ task, see section 2.2 and Figure 1, etc.). Segmenting the ground truth sentence, which participants imagined and subsequently typed from memory using a keyboard, into patches (see, e.g., Figure 3 A, etc.). The initial segment may serve as the context input, while the subsequent segment served as the continuation. Correspondingly, the fNIRS brain recordings associated with the continuation segment were extracted. In some examples, a 6 second (or other) delay may be applied to the fNIRS data, as identified by the FIR model (see, e.g., section 2.4, etc.). Processing the brain signals through a brain encoding model that maps the time series data to standard LLM embeddings. Further, in some embodiments, the context input may be simultaneously tokenized and converted into LLM embeddings. Concatenating the context input embeddings with the brain signal-generated embeddings, forming the prompt input to the LLM (see, e.g., Figure 3B, etc.). In some instances, during training, the brain encoding model may be trained to learn the mapping from brain signals to LLM embeddings, while the LLM may have its parameters frozen during this process. Generating, via the LLM, the continuation text based on the concatenated prompt input. Further, in some embodiments, the predicted continuation may then be compared with the ground truth continuation text to facilitate the training of the encoding model.

[0032] See, as one example, Figures 3A-3C, below, which are block diagrams showing the prompt tuning process in continuous imagined speech decoding. In the example of FIG. 3 A, the ground truth sentence, which the participant imagined and subsequently typed from memory, is segmented into context input and continuation. LLM embeddings are generated from the context input text, and the fNIRS brain recordings associated with the continuation segment goes through a brain encoding model to generate brain predicted LLM embeddings. The fire symbol represents the weights in the brain encoding model that are trainable.

[0033] Turning to the example of FIG. 3B, the brain signal -generated embeddings may be concatenated with the context input embeddings, forming the prompt input to LLM (here, e.g., in the example, a Llama2-7b LLM) with frozen parameters (snow symbol). The predicted embeddings may be converted to text and compared with the ground truth continuation text in order to generate a training loss. Permutation predictions are generated with permutation inputs, where brain signals corresponding to one sentence are paired with the context input from another sentence randomly chosen in the same participant. In the exemplary functionality of FIG. 3C,brain data from multiple participants, except the held-out test participant, are aligned using a ridge regression model to create a shared latent feature space, before being inputted to the brain encoding model training. In some example embodiments, the model may then be fine-tuned with, say, 100 trials and metrics are evaluated on, e.g., 200 held out test trials from the test participant. Data preparation

[0034] Consistent with the disclosed technology, various LLMs and LLM architecture / processing may be utilized. In the exemplary study above, for example, a Llama2-7b was used as the LLM for text processing and generation. An LLM works by leveraging a deep neural network, typically with billions of parameters, that has been trained on vast amounts of text data to predict and generate human-like text based on input text prompts. Such LLM(s) use this training to understand context, grammar, and semantics, allowing it to generate coherent and contextually appropriate responses.

[0035] In the example study and in various embodiments of the disclosed technology, each ground truth sentence imagined by the participant may be divided into segments. Given that such imagined sentences are in general short, each sentence may be segmented into two parts only (context input and continuation, see e.g., Figure 3A, etc.). Note that any typed sentence with three words or fewer and their related fNIRS data may be excluded from the continuous imagined speech decoder training, as they were considered too short for segmentation. In such exemplary embodiments, the texts may then be tokenized into token IDs with a max length of 32 with padding. These token IDs may then be converted to Llama2 embeddings afterwards.Additional custom tokens, e.g., <brain / > and < / brain>, etc. specific to each implementation, may be added to the tokenizer, which may be used to separate brain generated embeddings from the context input embeddings. The sample numbers of the brain signals corresponding to the continuation part of the sentence were extracted from the preprocessed fNIRS signals. In the example study, a fixed delay (e.g., of 6 second, etc.) was added to the brain data in order to account for BOLD delays (see FIR model section for delay estimation).

[0036] For permutation inputs, the continuation brain signals from a given sentence were paired with the context input text from another sentence randomly chosen from within the participant. This method kept the distribution of the participant’s brain signals compared to using random numbers or scrambled brain data. Note that permutation does not refer to the permutation of the time points or channels of the brain signals.

[0037] Additional experimental conditions of LLM+text and brain signal only may also be implemented. Here, for example, LLM+context refers to where the LLM uses only context input text for generating continuation predictions, and brain signal only refers to where brain signals alone are used for continuation predictions. In some embodiments, these conditions may follow the same data processing as described for brain and permutation input conditions mentioned above.

[0038] For all conditions of the exemplary study, the LLM embeddings for context input and continuation brain signals for all trials within a participant were stored in a Python pickle file, and were loaded as Pytorch datasets during training. The brain data was z-scored within each session for each participant in order to minimise the effects of different cap placement across different days of experiments.Brain encoding model

[0039] Consistent with the disclosed technology, various brain encoding models may be used. One such brain encoding model maps fNIRS data to LLM embeddings. Here for example, such model may takes a time-series input of shape [sequence length x number of channels], where sequence length is the number of time points in the brain signals segment corresponding to the continuation part of the sentence, and number of channels is the total number of fNIRS channels of 388.

[0040] In one example, known sequence-to-sequence (Seq2Seq) neural network model(s) with transformers may be used the brain encoding model. The Seq2Seq model is a versatile neural network capable of handling variable-length sequences effectively, compared to other simpler deep neural networks. For the exemplary study, a transformer model was used for both its encoder and decoder components. Transformers utilise a self-attention mechanism, which allows the model to weigh the importance of different time steps dynamically, regardless of their distance in the sequence. This capability is particularly advantageous for time series data such as fNIRS data, where important patterns and dependencies can occur over various time scales.

[0041] Further, in the example study, a transformer with a single encoder layer and a fully connected layer was used, with a fixed dropout rate of 0.3. As an encoder, it processes the input sequences of size 388 (number of fNIRS channels), with hidden size of 100. As a decoder, it takes the input sequence of size 100 from the encoder, and outputs sequences of shape 4096 (Llama2 embedding size). During the forward pass, the source sequence was first processed by the encoder, and the encoder's output was subsequently passed to the decoder to generate thefinal output sequence. The weights of the transformer model were initialised using Xavier uniform distribution for projection and linear weights, while setting biases to zero or a small constant, and applying similar initialization to the linear layers.Model training, validation and testing

[0042] In the example study, the training process involved two stages, a pretraining step and a main training step. Pretraining was run for 10 epochs before the main training started, with an initial learning rate of le-3. The pretraining loss computes the mean squared error (MSE) loss between the brain encoding model's predicted embeddings and the mean value of the ground truth continuation text embeddings. The ground truth texts were converted into embeddings, which were then fused into a mean representation to ensure a constant length. This approach also minimizes the probability of information leak about the ground truth text.

[0043] Further, during main training of the exemplary study, the parameters of the Llama2 LLM were frozen, preserving its inherent knowledge while utilizing its ability to generate coherent text outputs given limited input examples. Instead, the brain encoding model was being trained in an end-to-end pipeline in order to learn to generate LLM embeddings from brain signals. The training function initialized data loaders, where 200 trials were kept out for testing, the rest of the dataset were divided into 80% training and 20% validation. Training batch size was set at 8. Further, a known Adam optimizer with an initial learning rate of le-4 and a learning rate scheduler with step=l was used for training. The model weights are saved if a lower validation loss is achieved in the next epoch, and early stopping is implemented if the validation loss does not improve over 10 epochs. The max number of epochs was set to 50.

[0044] Training loss in main training of the example study was defined as the cross-entropy loss between the Llama2 predicted token logits and the true labels in the continuation text. LLM's predicted token logits were the raw, unnormalized scores output by the LLM for each token in the vocabulary, indicating the likelihood of each token being the next in the sequence, before applying a Softmax function to convert them into probabilities. The predicted and true sequences were aligned in lengths, and a mask extracted the portion of the predicted sequence corresponding to the true labels. The loss was then calculated using the cross-entropy function on the filtered logits and true labels, with the 'reduction' parameter set to "mean" to average the loss over the batch. This process ensures that the loss computation only considers the valid parts of the generated sequence, ignoring padding or other irrelevant tokens. In each training epoch, the gradients were reset to zero, the loss was backpropagated, and gradient norms were clipped to amaximum of 10 before updating the model parameters using the optimizer, with the total loss being accumulated for each batch.

[0045] Finally, the held out test trials were used for text generation and metric calculations. Predicted text tokens were generated given the concatenated prompt input from the context input and the brain predicted embeddings. The tokens were converted back to text using Llama2’s tokenizer, where special tokens such as < / s> and <s> are removed. The LLM predicted text and ground truth text for the continuation part of the sentence are then used as prediction and hypotheses for a series of metric calculations.Multi-participant alignment and fine tuning

[0046] Various different implementations of such systems and methods may be implemented by combining multiple participant’s data for training in order to maximise data usage, as well as increasing coverage of the overall semantic map (see, e.g., Figure 3C, etc.). Data from multiple participants except for the left out test participant was first aligned for training. This was accomplished, in the example study, by passing the brain data to a ridge regression model that maps the input from 388 channels to 100 features. The ridge regression model may comprise multiple linear layers, and processes data based on the given participant ID, so that each participant effectively has a different feature extractor and the resulting features reside in a latent feature space shared by all the training participants. This process aims to extract features and reduce noise from fNIRS signals. The extracted features are used as input to the brain encoding model instead of the full fNIRS signals.

[0047] Consistent with the disclosed technology, a leave-one-participant-out test may be used, where one participant’s data was left out and all other participants’ data was used for training. In the illustrative study, for example, before testing metrics were calculated, the pre-trained brain encoding model from multiple participants was first finetuned with 100 fine-tune trials from the test participants’ data. The data collection time for 1 trial is about 22s (7s for the imagined period, 15s for typing based on a typical typing speed of 40 words / min and the averaged sentence length of 10 words), which means the collection of 100 trials for fine tuning would have taken about 37 minutes to complete if collected in a separate run. The 200 test trials were held out from the finetuning process. The previously saved cache model was loaded and trained with the same procedures and parameters as mentioned in 2.5.4 with the finetune trials, then the test metrics are calculated on the held out test trials.Evaluation metrics

[0048] According to embodiments herein, natural language processing metrics may be used to evaluate model performance by comparing the model generated sentences to the ground truth sentences which the participant was imagining. These include BERTScores, a text evaluation metric that assesses the similarity between candidate and reference texts by computing the cosine similarity of their contextual embeddings derived from, e.g., the known BERT model. This is further divided into 3 scores - 1) BERT Fl, which combines precision and recall to measure accuracy by balancing both false positives and false negatives; 2) BERT P (Precision), a precision metric focusing on the correctness of the positive predictions made by the model; and3) BERT R (Recall), a recall metric assessing the model's ability to capture all relevant instances.4) BLEU-1 score, which measures the overlap of n-grams (uni grams) between the generated sentence and the reference, indicating surface similarity, a measure known in the art. 5) METEOR scores, which considers precision, recall, and synonymy, providing a more nuanced similarity score than BLEU, as known in the art. 6) ROUGE L score focuses on the longest common subsequence between the generated and reference sentences, reflecting structural similarity, as known in the art. 7) Word Error Rate (WER), which calculates the edit distance between the generated sentence and the reference, indicating accuracy, as known in the art. Consistent with the disclosed technology, utilization of one or more of these multiple metrics may provide a comprehensive evaluation of different aspects of model performance.Imagined Speech Detection Decoder

[0049] With the discrete sentence classification, i.e., referred to as MindGPT, and disclosed in parallel application(s), such technology may use an Extra Trees Classifier (XTC) model able to differentiate imagined speech from the rest condition with an average accuracy of 66% across four participants (best average accuracy: 71%). Such other imagined speech processing technologies however relied heavily on memory and participant training before data collection. Turning to the present disclisure, various similar / exemplary testing was repeated with the new fNIRS dataset of imagined speech data from four participants collected while performing the word cloud paradigm herein, to establish that imagined speech detection using fNIRS may be done via the presently-disclosed technology without relying on other cognitive functions or need for pacing imagined speech words at a specific word / minute rate using a metronome, removing any contamination of other cognitive functions, such as memory or auditory processing. Therefore, we trained a new XTC model to decode imagined speech from the rest data in fully preprocessed brain signals from this study. The same parameters and methods as with MindGPTwere used. All models were trained on within-participants to determine the best individual accuracy, although results from participants were also averaged to identify overall performance of our models in successfully classifying imagined speech vs rest condition.Imagined speech related brain activations

[0050] In the present exemplary testing / experimentation, in addition to applying decoding models to the imagined speech data, analyses involving haemodynamic response modelling and GLM- based statistical testing were conducted in order to show imagined speech related activations in the brain. To extend our knowledge of semantic representation in the brain, brain activations from the contrast ‘word cloud > rest’ were compared across participants.

[0051] In such testing, the fNIRS data was preprocessed to haemo data using the steps described in section 2.3. A first-level design matrix was constructed to model the hemodynamic response associated with neural activity, using the python package mne-nirs (version 0.6.0). Channels (distance > 10mm) are used in the general linear model, as known in the art, with a cosine function to model and correct for low-frequency drift in the signal, and a high-pass filter of 0.005Hz to remove slow signal variations not contributed to neuronal activities. The hemodynamic response function (HRF) model adopted was based on the Statistical Parametric Mapping (SPM) approach, a standard model for estimating the brain's vascular response to neural activity. Lastly, a stimulus duration of 7 seconds from the onset of each word cloud sentence (i.e. keyword highlight) and each resting period was implemented. Averaged short channel sequences (distance < 10mm) were included as nuisance regressors in the design matrix. Conditions ‘word cloud’ and ‘rest’ were specified and the GML parameters were estimated. The contrast 'word cloud > rest’ was then estimated from GLM theta values, and z-scores are calculated. Surface plots are generated with the estimated z-scores. Due to computing memory constraints, only 10 runs were used for each participant, totaling to around 50000 time points per participant. For group analysis, a group-level mixed linear model was run using the statsmodels package, fitting data with participant IDs as grouping, and the model was optimised using the "nm" method.

[0052] To verify the accuracy of this present methodology to localize brain areas recruited during the cognitive tasks and compare this accuracy in localization across participants, we also conducted finger tapping experiments at the start and end of each session and compared brain activation in response to right vs left finger tapping and their localization within the cortex. Right vs left finger tapping tasks have been previously shown to lead to a quite strong and clear differential activation in the contralateral motor cortex, with respect to the hand completing thefinger tapping, as known in the art. Therefore, identifying the correct localisation of brain activation during the finger tapping experiment would confer us with increased confidence with respect to precise cap placement and brain region coverage, as well as good quality of data collected.Experimental Proofs / ResultsDataset statistics

[0053] Statistics of each participant's total number of topics and sentences completed for the word cloud tasks are summarised in Table 1. Participants completed on average 216 topics in the word cloud task, although the number of topics varied across participants. The average number of words per sentence also varied across participants, where participant 4 had on average the longest imagined sentence length of 10.5 words, while participant 3 had the shortest averaged sentence lengths of 8.67 words. However, participant 1 used the most unique words per sentence of 3.28 words compared to other participants. Summarizing language styles using statistics is challenging; however, participants exhibited varied writing styles in addition to differences in sentence lengths and unique word choices. The imagined sentences exhibit a more conversational and informal tone (see Table 3 for examples), characteristic of spoken language, rather than the structured and formal style typical of Wikipedia-style text, as known in the art.Table 1. Statistics on total number of sentences, topics and keywords imagined by the participants during the data collectionIdentifying BOLD delays with FIR models

[0054] To determine the optimal BOLD delay to use in the imagined speech decoding model training, a FIR model was employed to estimate coefficients across a range of delays. FIR models do not assume a fixed haemodynamic response function, allowing for the estimation of individual haemodynamic responses and accommodating inter-individual differences. The delaywith the highest coefficient from these estimations is considered the optimal delay. Figure 4 illustrates the delay coefficients from a group estimation for all runs within each participant, categorised by word cloud and resting conditions, as well as oxy / deoxygenated haemoglobin (HbO / HbR). For participants 1, 2, and 3, the highest coefficient for HbO occurred at the third delay (equivalent to a delay of 4-6 seconds, given a resampling rate of 0.5 Hz), while for participant 4, it was at the fourth delay (6-8 seconds). The second highest delay for participants 1 and 3 was 6-8 seconds. Based on these findings, a delay of 6 seconds is applied to the fNIRS data to accurately account for the BOLD responses in word cloud tasks. This standardized delay was chosen to maintain consistency across participants and simplify the model implementation, ensuring reliable comparison of results.Imagined speech decoder: individual participant test results

[0055] According to the illustrative example herein, Table 2 below shows test metrics for imagined speech decoders trained and tested with individual data. The metrics that measure both exact and semantic similarities between the predicted continuation texts as hypotheses, and the ground truth continuation from typed imagined sentences as references. The table heading represent 4 different experimental conditions - 1) LLM+context, where only context input text inputs are used for the LLM to generate continuation predictions; 2) Brain signal only, where brain signals without context input are used for continuation predictions; 3) Brain+context, where brain signals are converted to LLM embeddings and concatenated to context input embeddings, before being inputted to LLM for continuation predictions (the prompt tuning approach); 4) Permutation+context, where brain signals from another permuted trial are paired with context input from the current trial for continuation generation. This condition tests specifically whether the prompt tuning approach with unrelated but distribution conforming brain data improves the quality of generated continuations, acting as a stringent control condition to 3). The table shows both metric values and the corresponding t-test p-values between brain and permutation inputs (experiments 3 and 4). The metrics that are higher for brain compared to permutation inputs in values are highlighted in blue and those that are statistically significant (t- test p<0.05) are highlighted in green and provided in bold.

[0056] Using either only the context inputs or the brain signals produced generally low test metrics (experiments 1 and 2 in Table 2), however, by combining context inputs with corresponding brain signals, there are sizable improvements across all test metrics. Notably, when comparing generated continuation from brain and permutation inputs with the ground truthcontinuation (experiments 3 and 4 in Table 2), the exact match metric (B LEU-1) are statistically significantly higher for brain compared to permutation input for 3 out of 4 participants, and the semantic similarity metric (BERT P) are statistically significantly higher for 2 out of 4 participants with brain inputs, with an additional participant approaching significance. For participant 1, BERT P (p = 0.02) and BLEU-1 (p = 0.004) showed significant improvements (see definition in section 2.5.2 Data preparation). For participant 2, BLEU-1 (p < 0.001) and METEOR (p = 0.017) demonstrated significant enhancements, while BERT P (p = 0.09) approached significance, indicating a trend towards improved performance. For participant 4, BERT Fl (p = 0.05), BERT P (p = 0.01), and BLEU-1 (p = 0.027) were significantly better than the permutation. As for participant 3, the metrics BLEU-1, METEOR, ROUGE-L and WER are better than permutation but did not reach statistical significance.

[0057] Overall, these results highlight the capability of the disclosed technology’s prompt tuning-based continuous imagined speech decoder to leverage brain signals for generating text continuations that are both semantically and exactly similar to the ground truth. The consistent performance across multiple participants, as evidenced by significant improvements in key metrics, shows the robustness and potential of our innovative systems and methods of decoding brain signals for continuous text generation tasks.Table 2. Individually trained imagined speech decoders results.Metrics measure exact and semantic similarity between predicted continuation against ground truth continuation texts for 200 test sentences. Numbers in brackets represent different experimental conditions. P-values are calculated with t-tests. A fixed delay of 6s is applied to all participants’ data, (blue - brain input has outperformed the permutation input, green - brain input has outperformed the permutation input and is statistically significant with p<0.05)

[0058] Table 3 provides example decoder-generated continuation sentences using both brain and permutation inputs (experiments 3 and 4), along with the context input and ground truth continuation texts. The context input represents the initial part of the imagined sentence. The LLM is then guided by either the brain inputs or the permutation inputs (which consists of brain data from another sentence) to complete the prediction. These examples are chosen because the brain predictions had higher BLEU-1 scores than the permutation predictions. Please note that these examples therefore represent best examples of successful prompt tuning decoding.

[0059] The example of Table 3 illustrates that continuations generated from true brain data are closer to the ground truth (when metrics are statistically significant) in terms of both precise wording and semantic meaning. This trend is evident for participants 1, 2, and 4. In contrast, participant 3 had significantly fewer such sentences, consistent with their test metrics, which showed similar performance between brain and permutation predictions. This demonstrates the effectiveness of using brain data for generating text continuations closer to the actual imagined contents.Table 3. Predicted continuation examples from individually trained imagined speech decoders described above. The examples are from test cases that are held out in training.Imagined speech decoder: multi-participant alignment and finetune results

[0060] Table 4 presents the test metrics and statistical significance for the multi-participant alignment and fine-tuning experiment. To address time constraints that prevent participants from covering all topics, combining data from multiple participants can enhance the quantity and diversity of training topics. A simple ridge regression linear model aligns each participant's data to a unified latent feature space shared by all training participants, which is then used as input to the brain encoding model and training procedures described previously. Fine-tuning was conducted with 100 trials from the held-out test participant, followed by test metrics calculation using another 200 left out test trials. The same 4 experimental conditions were used as described in the previous section.

[0061] The results revealed indicate that participants 2 and 3 achieved significantly higher BERT metrics (precision, recall, and Fl score) when using brain over permutation inputs (experiments 3 and 4), while other metrics were higher but did not reach statistical significance. Participant 3, who did not achieve statistically significant performance when training an individual decoder with their own data, showed improved results when using the model trained with data from other participants. This suggests that combining data from multiple participants potentially enhances the model's ability to generalize across semantic meanings. However, participants 1 and 4 did not exhibit any statistically significant results comparing brain against permutation inputs. Given that participants 1 and 4 had distinct imagined sentence statistics (participant 1 used the most unique words per sentence, and participant 4 had the longest average imagined sentence length) (see section 3.1 for dataset statistics per participant), it is possible that the fine-tuning process couldnot fully mitigate these variations. For completeness, LLM+context and brain signals only conditions were tested (experiments 1 and 2), which showed lower test metrics compared to combined inputs, consistent with results from individually trained decoders. Overall, while multi-participant alignment training combined with fine-tuning shows potential for building a more versatile model, it may still be influenced by individual differences.Table 4. Multi-participant aligned decoder results. The decoder was trained on all other participants’ data, and then fine tuned and tested on the test participant’s data.Metrics measure exact and semantic similarity between predicted continuation against ground truth continuation texts for 200 test sentences. Numbers in brackets represent different experimental conditions. P-values are calculated with t-tests. A fixed delay of 6s is applied to all participants’ data, (blue - brain input has outperformed the permutation input, green - brain input has outperformed the permutation input and is statistically significant with p<0.05)Imagined speech detection

[0062] Our XTC classification model showed above chance results in distinguishing brain signals from imagined speech from rest (see, e.g., Figure 5, etc.). A total of 386, 532, 706, and 710 trials were included for Participant 1, 2, 3, and 4, respectively. The total number of trials consisted of an equivalent number of imagined speech and rest trials (i.e., imagined speech trials = ’A of total trials, rest trials = ’A of total trials). Please note that the dataset was balanced to ensure the same number of imagined speech and rest trials. Overall, our XTC model achieved an average accuracy of -76% (p < 0.001, chance: 50%) when considering averaged accuracies across folds for the 4 subjects included in this test (max accuracy across subjects: 78%, p < 0.001) . Our best participant (participant 2) reported a best average accuracy of -88% across the 3 folds in cross validation (p < 0.001) and max accuracy of -90% (p < 0.001). The classifier was found to perform worse on participants 1 and 4, although it still performed well above chance (p < 0.001).Brain areas underlying imagined speech

[0063] Figure 6 illustrates the surface cortex plots of HbO activations of the contrast ‘word cloud > rest’ for all four participants included in this study. Stimulus durations of 7s from each word cloud keyword highlight onset was considered. The colour bar represents the z-score values of the contrast. This analysis identified a recurrent pattern of multiple brain regions recruited during imagined speech, including the lateral temporal cortex, the dorsolateral prefrontal cortex (DLPFC) and the visual processing areas in the occipital region, as known in the art. Confirmation of these regions further prove that high density fNIRS is capable of capturing imagined speech processing in the brain, and that the disclosed technology such as the word cloud features and functionality can effectively prompt imagined speech by removing the confound of memorization and recall.Overall Aspects of the Disclosed Technology

[0064] According to some aspects of the disclosed technology, a high-density fNIRS-based BCI system may be implemented for deciphering imagined sentences as well as their semantic content. Here, for example, instead of classifying a limited number of sentences, systems andmethods herein implement an open vocabulary, continuous decoder for imagined speech fNIRS data. Consiste with the present inventions, various novel technologies (e.g., including preprocessing and reformatting data for training models, model training and hyperparameter optimization, among others) involving collecting and processing high-density fNIRS data on imagined speech. Here, for example, implementations may prompt participants with topic words and keywords and have them imagine sentences. Ground truth sentences may be obtained by having participants typing them out on a computer using a keyboard after the imagined period. Aspects may also involve producing and / or using a prompt tuning-based model for generating continuous natural text from brain signals. Utilizing, in one exemplary implementation, e.g., of the study, the LLM Llama2-7b with a custom brain encoding model, brain signal guided inputs demonstrated improved language and semantic similarity in generated text continuation compared to permutation conditions. For individual training, the model with brain inputs achieved statistically significant higher BLEU-1 scores in 3 out of 4 participants as compared with permutation inputs, and significantly higher BERT P scores in 2 out of 4 participants (an additional one approaching significance). For multi-participant alignment, the model trained on multiple participants’ data and fine tuned on the held out participant’s data achieved statistically significant higher BERT Fl / P / R scores in 2 out of 4 participants with brain compared to permutation inputs. Additionally, the disclosed technology was able to surpass our previous results on imagined speech detection from rest and achieved a 10% increase in decoder accuracy, and identified brain activations consistent with imagined speech research. Our innovations establish that using high-density fNIRS together with sophisticated artificial intelligence methods like LLM and prompt tuning, consistent with the disclosed technology / innovations, enables development and achievement of effective human-to-digital communication systems, novel results achieved via the disclosed technology.

[0065] Imagined speech paradigms in previous studies have predominantly focused on single words and may not be optimal for data collection aimed for developing an open-vocabulary continuous imagined speech decoder. The reasons include their limited vocabulary or the absence of a time-bound ground truth, especially when participants recite long paragraphs from memory. To address these limitations, systems and methods herein may include or involve word cloud technology / processing and / or other features. Such new technology may include prompting participants to imagine new sentences within a specific time window, allowing each ground truth sentence to be accurately aligned with the corresponding section of brain data. The topic-basedblocks enabled participants to easily recall the imagined sentences after the imagination period concluded. Additionally, the recall and typing activities had minimal impact on brain activity during the imagination phase. Crucially, this design eliminates the confounding effects of memory recall or auditory processing when using a metronome for pacing during the imagination phase, which can obscure brain activity related to imagined speech and hinder decoder performance. Moreover, such word cloud technology is also expandable with additional topics of varying semantic meanings and, advantageously, requires minimal training for new participants. Thus, the disclosed technology is a novel and valuable improvement in the field of existing imagined speech paradigms.

[0066] As set forth in the present disclosure, several distinct improvements may be implemented to optimize embodiments for high-density fNIRS data. Firstly, systems and methods herein may apply a finite impulse response (FIR) model to identify the appropriate BOLD delay specifically for the word cloud task. Unlike most fMRI decoding studies that use a fixed BOLD delay (as known in the art), this customization is highly advantageous for fNIRS signals, which generally have a weaker signal-to-noise ratio (SNR) compared to fMRI. Further, as also known, offsets in delays can significantly impact decoding performance. The disclosed technology establishes that the FIR model derived BOLD delays differed from conventional values (e.g., 8 seconds used in some other known art) and produced better results. Secondly, systems and methods herein may utilize modified architecture for the brain encoding model. One illustrative embodiment, for example, utilizes a Seq2Seq model with transformers as the encoder and decoder components. This technical solution leverages the self-attention mechanism of transformers and the contextual understanding of Seq2Seq models to dynamically model different time steps, capturing temporal dependencies more effectively than existing knowledge in the art. Such technical improvements / adaptations are particularly beneficial for fNIRS data, which has a higher temporal resolution compared to fMRI data. Finally, the disclosed technology is further substantiated via a multiparticipant alignment and fine-tuning experiment to demonstrate that combining data from multiple participants can potentially enhance the decoder’s performance. Various important supplementations and improvements were added and involved on top of straightforward ridge regression model to align multiple participants’ brain data into a shared latent feature space before inputting it to the brain encoding model. The innovations of the overall, improved technology showed improvements in test metrics, particularly for participant 3, who did not show better results for brain inputs when training with their own data alone. Finally, various

[0067] Improvements and enhancements in the present, innovative prompt tuning model - including more accurate BOLD delay estimation, Seq2Seq transformer architecture for the brain encoding model, and multi-participant data alignment - significantly improve continuous imagined speech decoding from high-density fNIRS data. Both the individually trained participant test results and the multi-participant alignment and fine-tuning outcomes highlight the advantages of decoding continuous imagined speech from brain signals. For individually trained decoders, significant improvements in key test metrics such as BLEU-1 and BERT P for a majority of test participants in individual training highlight the effectiveness of the prompt tuning-based text generation method. Improvements may also be seen in the multi-participant alignment and fine-tuning experiment, where brain inputs for various participants showed significantly higher BERT scores that indicate semantic similarity only.

[0068] Further, various systems and methods herein may may be implemented with an XTC model to differentiate between imagined speech and resting states, with the example study achieving an average accuracy of 76% across four subjects. These findings corroborate the innovative aspects of the disclosed technology, showing a 10% improvement in decoding accuracy between imagined speech and resting states in the current study, which may be attributable to the increased engagement in imagined speech and enhanced data quality from the word cloud paradigm. Further, the increased visual region activations from keyword highlighting may also contribute to the improved decoding accuracy. Brain activation analysis additionally revealed significant brain activations in regions such as the dorsolateral prefrontal cortex (DLPFC), lateral temporal cortex, and visual areas when contrasting imagined speech with rest, further support the benefits and advantages of the disclosed technology.

[0069] Overall, the disclosed technology sets forth herein enables various systems and methods involving an open-vocabulary, continuous decoder for imagined speech using high-density fNIRS data as well as other aspects of developing and implementing such technology. According to certain embodiments, a novel word cloud paradigm may be implemented, which improved data quality and allowed participants to generate imagined sentences with varied semantic meanings. Further, aspects such as the prompt tuning-based innovations for text generation guided by imagined speech brain signals are strong advances. Significant improvements in key metrics, such as BLEU-1 and BERT P scores, were observed during testing, demonstrating the effectiveness of the disclosed technology. Moreover, innovations herein involved with the multiparticipant alignment and fine-tuning experiment further establish that the disclosed techniquesfor combining data from multiple participants enhances decoder performance, with statistically significant improvements observed in BERT scores for test participants.Various Implementations of the Disclosed Technology

[0070] While the above disclosure sets forth certain illustrative examples, the present disclosure encompasses multiple other potential arrangements and components that may be utilized to achieve the brain-computer interface innovations of the disclosed technology. Some other such alternative arrangements and / or components may include or involve other optical architectures that provide the desired results, signals, etc., while some such implementations may also enhance performance, correctness, and / or other metrics further.

[0071] While the above disclosure sets forth certain illustrative examples, such as embodiments utilizing, involving and / or producing fast optical signal (FOS) and haemodynamic (e.g., NIRS, etc.) brain-computer interface features, the present disclosure encompasses multiple other potential arrangements and components that may be utilized to achieve the brain-interface innovations of the disclosed technology. Some other such alternative arrangements and / or components may include or involve other optical architectures that provide the desired results, signals, etc. (e.g., pick up NIRS and FOS simultaneously for brain-interfacing, etc.), while some such implementations may also enhance resolution and other metrics further.

[0072] Among other aspects, for example, implementations herein may utilize different optical sources, neuroimaging modality altogether, Al system to interact with and Al model to decode the brain data than those set forth, above. Here, for example, such optical sources may include one or more of: semiconductor LEDs, superluminescent diodes or laser light sources with emission wavelengths principally, but not exclusively within ranges consistent with the near infrared wavelength and / or low water absorption loss window (e.g., 700-950nm, etc.); nonsemiconductor emitters; sources chosen to match other wavelength regions where losses and scattering are not prohibitive; here, e.g., in some embodiments, around 1060nm and 1600nm, inter alia; narrow linewidth (coherent) laser sources for interferometric measurements with coherence lengths long compared to the scattering path through the measurement material (here, e.g., (DFB) distributed feedback lasers, (DBR) distributed Bragg reflector lasers, vertical cavity surface emitting lasers (VCSEL) and / or narrow linewidth external cavity lasers; coherent wavelength swept sources (e.g., where the center wavelength of the laser may be swept rapidly at 10-200 KHz or faster without losing its coherence, etc.); multi -wavelength sources where a single element of co packaged device emits a range of wavelengths; modulated sources (e.g.,such as via direct modulation of the semiconductor current or another means, etc.); and pulsed laser sources (e.g., pulsed laser sources with pulses between picoseconds and microseconds, etc.), among others that meet suffi ci ent / pro scribed criteria herein.

[0073] Implementations herein may also utilize different optical detectors than those set forth, above. Here, for example, such optical detectors may include one or more of: semiconductor pin diodes; semiconductor avalanche detectors; semiconductor diodes arranged in a high gain configuration, such as transimpedance configuration(s), etc.; single-photon avalanche detectors (SPAD); 2-D detector camera arrays, such as those based on CMOS {complementary metal oxide semiconductor} or CCD {charge-coupled device} technologies, e.g., with pixel resolutions of 5x5 to 1000x1000; 2-D single photon avalanche detector (SPAD) array cameras, e.g., with pixel resolutions of 5x5 to 1000x1000; and photomultiplier detectors, among others that meet sufficient / proscribed criteria herein.

[0074] Implementations herein may also utilize different optical routing components than those set forth, above. Here, for example, such optical routing components may include one or more of: silica optical fibre routing using single mode, multi-mode, few mode, fibre bundles or crystal fibres; polymer optical fibre routing; polymer waveguide routing; planar optical waveguide routing; slab waveguide / planar routing; free space routing using lenses, micro optics or diffractive elements; and wavelength selective or partial mirrors for light manipulation (e.g. diffractive or holographic elements, etc.), among others that meet sufficient / proscribed criteria herein.

[0075] Implementations herein may also utilize other different optical and / or computing elements than those set forth, above. Here, for example, such other optical / computing elements may include one or more of: interferometric, coherent, holographic optical detection elements and / or schemes; interferometric, coherent, and / or holographic lock-in detection schemes, e.g., where a separate reference and source light signal are separated and later combined; lock in detection elements and / or schemes; lock in detection applied to a frequency domain (FD) NIRS; detection of speckle for diffuse correlation spectroscopy to track tissue change, blood flow, etc. using single detectors or preferably 2-D detector arrays; interferometric, coherent, holographic system(s), elements and / or schemes where a wavelength swept laser is used to generate a changing interference patter which may be analyzed; interferometric, coherent, holographic system where interference is detected on, e.g., a 2-D detector, camera array, etc.; interferometric, coherent, holographic system where interference is detected on a single detector; controllablerouting optical medium such as a liquid crystal; and fast (electronics) decorrelator to implement diffuse decorrelation spectroscopy, among others that meet sufficient / proscribed criteria herein.

[0076] Implementations herein may also utilize other different optical schemes than those set forth, above. Here, for example, such other optical schemes may include one or more of: interferometric, coherent, and / or holographic schemes; diffuse decorrelation spectroscopy via speckle detection; FD-NIRS; and / or diffuse decorrelation spectroscopy combined with TD-NIRS or other variants, among others that meet sufficient / proscribed criteria herein.

[0077] Implementations herein may also utilize other multichannel features and / or capabilities than those set forth, above. Here, for example, such other multichannel features and / or capabilities may include one or more of: the sharing of a single light source across multiple channels; the sharing of a single detector (or detector array) across multiple channels; the use of a 2-D detector array to simultaneously receive the signal from multiple channels; multiplexing of light sources via direct switching or by using “fast” attenuators or switches; multiplexing of detector channels on to a single detector (or detector array) via by using “fast” attenuators or switches in the routing circuit; distinguishing different channels / multiplexing by using different wavelengths of optical source; and distinguishing different channels / multiplexing by modulating the optical sources differently, among others that meet sufficient / proscribed criteria herein.

[0078] As disclosed herein, implementations and features of the present inventions may be implemented through computer-hardware, software and / or firmware. For example, the systems and methods disclosed herein may be embodied in various forms including, for example, one or more data processors, such as computer(s), server(s), and the like, and may also include or access at least one database, digital electronic circuitry, firmware, software, or in combinations of them. Further, while some of the disclosed implementations describe specific (e.g., hardware, etc.) components, systems, and methods consistent with the innovations herein may be implemented with any combination of hardware, software and / or firmware. Moreover, the above-noted features and other aspects and principles of the innovations herein may be implemented in various environments. Such environments and related applications may be specially constructed for performing the various processes and operations according to the inventions or they may include a general-purpose computer or computing platform selectively activated or reconfigured by code to provide the necessary functionality. The processes disclosed herein are not inherently related to any particular computer, network, architecture, environment, or other apparatus, andmay be implemented by a suitable combination of hardware, software, and / or firmware. For example, various general-purpose machines may be used with programs written in accordance with teachings of the inventions, or it may be more convenient to construct a specialized apparatus or system to perform the required methods and techniques.

[0079] In the present description, the terms component, module, device, etc. may refer to any type of logical or functional device, process or blocks that may be implemented in a variety of ways. For example, the functions of various blocks may be combined with one another and / or distributed into any other number of modules. Each module may be implemented as a software program stored on a tangible memory (e.g., random access memory, read only memory, CD- ROM memory, hard disk drive) within or associated with the computing elements, sensors, receivers, etc. disclosed above, e.g., to be read by a processing unit to implement the functions of the innovations herein. Also, the modules may be implemented as hardware logic circuitry implementing the functions encompassed by the innovations herein. Finally, the modules may be implemented using special purpose instructions (SIMD instructions), field programmable logic arrays or any mix thereof which provides the desired level performance and cost.

[0080] Aspects of the systems and methods described herein may be implemented as functionality programmed into any of a variety of circuitry, including programmable logic devices (PLDs), such as field programmable gate arrays (FPGAs), programmable array logic (PAL) devices, electrically programmable logic and memory devices and standard cell-based devices, as well as application specific integrated circuits. Some other possibilities for implementing aspects include: memory devices, microcontrollers with memory (such as EEPROM), embedded microprocessors, firmware, software, etc. Furthermore, aspects may be embodied in microprocessors having software-based circuit emulation, discrete logic (sequential and combinatorial), custom devices, fuzzy logic, neural networks, other Al (Artificial Intelligence) or machine learning systems, quantum devices, and hybrids of any of the above device types.

[0081] Other implementations of the inventions will be apparent to those skilled in the art from consideration of the specification and practice of the innovations disclosed herein. It is intended that the specification and examples be considered as exemplary only, with a true scope and spirit of the inventions being indicated by the present disclosure and various associated principles of related patent doctrine.

[0082] As one overview of aspects of the disclosed technology, systems, methods and wearable devices associated with mind / brain-computer interfaces are disclosed. Embodiments herein include features related to one or more of optical-based brain signal acquisition, decoding modalities, encoding modalities, brain-computer interfacing, AR / VR content interaction, brain state assessment, signal to noise ration enhancement, and / or motion artefact reduction, among other features set forth herein. Certain implementations may include or involve processes of collecting and processing brain activity data, such as those associated with the use of a braincomputer interface that enables, for example, decoding and / or encoding a user’s brain functioning, neural activities, and / or activity patterns associated with thoughts, including sensory-based thoughts. Further, the present systems and methods may be configured to leverage brain-computer interface and / or non-invasive wearable device aspects to provide enhanced user interactions for next-generation wearable devices, controllers, and / or other computing components based on the human thoughts, brain signals, and / or mind activity that are detected and processed.

[0083] It should also be noted that various logic and / or features disclosed herein may be enabled using any number of combinations of hardware, firmware, and / or as data and / or instructions embodied in various machine-readable or computer-readable media, in terms of their behavioral, register transfer, logic component, and / or other characteristics. Computer-readable media in which such formatted data and / or instructions may be embodied include, but are not limited to, non-volatile storage media in tangible various forms (e.g., optical, magnetic or semiconductor storage media), though do not encompass transitory media.

[0084] Additionally, while described in terms of specific software approaches above, other software implementations will be apparent to those skilled in the art from consideration of the specification and practice of the innovations disclosed herein. For example, another implementation of the systems described herein may include one or more CNNs, ridge regression models, etc., and / or may use transformers or other neural networks to decode brain data (e.g., imaged speech sentences, etc.). In some embodiments, the systems described herein may incorporate additional and / or alternative neuroimaging methodologies to extra brain data including but not limited to EEG, fNIRS and EEG together, MEG or fMRI, and / or even with more invasive neuro-signal acquisition techniques and methodologies. The Appendices information submitted herewith describe some additional example implementations and technologies that may be implemented with and / or involved in some embodiments, e.g., toprovide brain signals, perform signal and / or data processing and / or otherwise be utilized in or with the disclosed technology.

[0085] Other implementations of the disclosed technology / present inventions will be apparent to those skilled in the art from consideration of the specification and practice of the innovations disclosed herein. It is intended that the specification and examples be considered as exemplary only, with a true scope and spirit of the inventions being indicated by the present disclosure and various associated principles of related patent doctrine.APPENDIX TO SPECIFICATION - COLOR TABLES FROM THE SPECIFICATIONTable 2. Individually trained imagined speech decoders results.Metrics measure exact and semantic similarity between predicted continuation against ground truth continuation texts for 200 test sentences. Numbers in brackets represent different experimental conditions. P-values are calculated with t-tests. A fixed delay of 6s is applied to all participants’ data, (blue - brain input has outperformed the permutation input, green - brain input has outperformed the permutation input and is statistically significant with p<0.05)Table 4. Multi-participant aligned decoder results. The decoder was trained on all other participants’ data, and then fine tuned and tested on the test participant’s data.Metrics measure exact and semantic similarity between predicted continuation against ground truth continuation texts for 200 test sentences. Numbers in brackets represent different experimental conditions. P-values are calculated with t-tests. A fixed delay of 6s is applied to all participants’ data, (blue - brain input has outperformed the permutation input, green - brain input has outperformed the permutation input and is statistically significant with p<0.05)

Claims

Claims:

1. A computer-implemented method for processing and / or decoding neural activity corresponding to speech, the method comprising: presenting to a user one or more topic cues, in response to which the user silently imagines a sentence of arbitrary vocabulary; acquiring, with at least one non-invasive neuro-sensing modality, time-series neural data that temporally overlap the user’s imagination of the sentence; processing and / or encoding the neural data with at least one brain-encoder model to produce intermediate data compatible with a large language model (LLM); performing processing that generates text that represents the user-imagined sentence; and generating an output and / or a control signal based on the text.

2. The method of claim 1 or the invention of any claim herein, wherein the processing and / or encoding the neural data includes encoding the neural data with a brain-encoder model to produce, as at least part of the intermediate data, one or more latent vectors that reside in a representation space compatible with a large language model (LLM); processing concatenating or otherwise fusing the latent vectors with embeddings of the text context, if any, to form a composite prompt; providing the composite prompt to the frozen or partially-frozen LLM, thereby causing the LLM to autoregressively generate a continuation portion of text that represents the user- imagined sentence; and / or outputting the generated text and / or a control signal derived therefrom.

3. The method of claim 1 or the invention of any claim herein, further comprising: generating, as the output and / or control signal, textual output directly from neural activity corresponding to speech, such as, preferably, self-generated (imagined) speech.

4. A computer-implemented method for generating textual output directly from neural activity corresponding to speech, such as self-generated (imagined) speech, the method comprising: a. presenting to a user one or more topic cues that induce the user to silently imagine a sentence of arbitrary vocabulary; b. acquiring, with at least one non-invasive neuro-sensing modality, time-series neural data that temporally overlap the user’s imagination of the sentence; c. optionally, receiving text context;d. encoding the neural data with a brain-encoder model to produce one or more latent vectors that reside in a representation space compatible with a large language model (LLM); e. concatenating or otherwise fusing the latent vectors with embeddings of the text context, if any, to form a composite prompt; f. providing the composite prompt to the frozen or partially-frozen LLM, thereby causing the LLM to autoregressively generate a continuation portion of text that represents the user-imagined sentence; and g. outputting the generated text and / or a control signal derived therefrom.

5. The method of any preceding claim or the invention of any claim herein, further comprising: receiving the text context, wherein the text context is selected from (i) no context - just the brain data alone, (ii) a topic cue alone, and / or (iii) a context-input portion of the sentence obtained by having the user type or otherwise supply the sentence after imagination.

6. The method of any preceding claim or the invention of any claim herein, wherein the neurosensing modality is selected from functional near-infrared spectroscopy (fNIRS), functional magnetic-resonance imaging (fMRI), magneto-encephalography (MEG), electroencephalography (EEG), high-density diffuse optical tomography (HD-DOT), or any combination thereof.

7. The method of claim 4 or the invention of any claim herein, wherein step c implements a selectable context policy such as one or more of:(i) brain-only prompting with no text context,(ii) topic-only prompting using a cue word or phrase, and(iii) context + brain prompting that supplies both typed context input and neural embeddings, thereby constraining the LLM’s output according to task requirements.

8. The method of claim 7 or the invention of any claim herein, wherein step c and / or the selectable context policy is utilized / implemented in real-time, such as, preferably, via using feedback from a GUI, and / or supplying varying levels of context to the model.

9. The method of any preceding claim or the invention of any claim herein, further comprising: a. training the brain-encoder model with paired neural-text examples collected from multiple users; and b. fine-tuning the trained model on a limited set (for example < 100 trials) of user-specific data to produce a participant-specific decoder.

10. The method of claim 9 or the invention of any claim herein, wherein training includes aligning data from different users into a shared latent feature space via a linear or non-linear mapping prior to fine-tuning.

11. The method of any preceding claim or the invention of any claim herein, wherein the brainencoder model comprises one or more of a sequence-to-sequence transformer, convolutional neural network, recurrent network, tree-based ensemble, regression model, or a hybrid thereof, such as, optionally, trained with prompt-tuning or adapter techniques while LLM parameters remain fixed; and / or wherein, preferably, the LLM parameters are unfrozen to improve performance.

12. The method of any preceding claim or the invention of any claim herein, further comprising: estimating, for each user or session, a haemodynamic delay by fitting a fmite-impulse- response (FIR) model to the neural data and time-shifting the data accordingly before step d.

13. The method of any preceding claim or the invention of any claim herein, wherein steps b-g are performed continuously or in near real-time with an end-to-end latency not exceeding 30 seconds including the imagination window.

14. A system comprising: a. a wearable or portable neuro-sensing apparatus configured to operate consistent with any one or more of the claims herein; b. at least one processor configured to execute steps d-g of claim 2; and c. a communication interface that delivers the generated text or control signal to an external computing service or Al agent, thereby establishing a direct thought-to-machine channel.

15. A system comprising: one or more computer processors; one or more non-transitory computer readable media, the non-transitory computer readable media including computer-readable program instructions that, upon execution by the one or more computer processors, cause the one or more computer processors to perform operations including: one or more of steps d-g of claim 4; and at least one communication interface that delivers a generated text or control signal to an external computing service or Al agent, thereby establishing a direct thought-to-machine channel.

16. The system of claim 15 or the invention of any claim herein, further comprising:a wearable or portable neuro-sensing apparatus configured to operate consistent with any one or more of the claims herein.

17. A system comprising: one or more computer processors; one or more non-transitory computer readable media, the non-transitory computer readable media including computer-readable program instructions that, upon execution by the one or more computer processors, cause the one or more computer processors to perform operations including: one or more aspects, steps, features, and / or functionality recited in any claim here or set forth elsewhere in the present disclosure.

18. The system of claim 17 or the invention of any claim herein, further comprising: a wearable or portable neuro-sensing apparatus configured to operate consistent with any one or more of the claims herein.

19. The system of claim 17 or the invention of any claim herein, further comprising: at least one communication interface that delivers a generated text or control signal to an external computing service or Al agent.

20. The system of claim 19 or the invention of any claim herein, wherein, as a function of the at least one communication interface transmitting the generated text or control signal to the eternal computing service or the Al agent, a direct thought-to-machine communication channel is established.

21. A computer-implemented method of decoding imaged sentences through a non-invasive neural interface, the method comprising: a user wearing a neural interface; detecting, via a neural interface, a sentence imagined by a user; processing, via the at least one processor, the sentence to break the sentence down into context input and continuation, wherein the context input is tokenized; extracting neural data signals from the continuation; processing the neural data signals to create brain encoding model, wherein the brain encoding model is used to create brain predictions to be embedded with tokenize context input to create a prompt input; entering prompt input into a large language model; creating predictions from the large language model;creating training data from the predictions; combining said training data with other user’s training data wherein said combined data is used for participant-specific mapping; and outputting test metrics utilizing a brain encoding model based on the participant-specific mapping.

22. The method of claim 21 or the invention of any claim herein, wherein the neural data signals comprise functional near-infrared spectroscopy (fNIRS) data.

23. The method of any claim above or the invention of any claim herein, wherein the prediction comprises baseline predicted continuation and / or brain predicted continuation, wherein the baseline predicted continuation information and / or the brain predicted continuation information are used to create the training data.

24. The method of any claim above or the invention of any claim herein, wherein the combined data is used for participant-specific linear mapping.

25. The method of claim 24 or the invention of any claim herein, further comprising: creating a ridge regression model from said linear mapping.

26. The method of claim 25 or the invention of any claim herein, further comprising: transforming the ridge regression model to create shared latent features.

27. The method of claim 26 or the invention of any claim herein, further comprising: inputting shared latent features into the brain encoding model.

28. The method of claim 27 or the invention of any claim herein, further comprising: separating user’s data into isolation data.

29. The method of claim 28 or the invention of any claim herein, wherein the isolated data includes at least one finetune set and at least one test set.

30. The method of claim 29 or the invention of any claim herein, further comprising: finetuning the brain encoding model based on the at least one finetune set.

31. The method of claim 30 or the invention of any claim herein, further comprising: testing the brain encoding model with the at least one test set.

32. The method of claim 31 or the invention of any claim herein, further comprising: once the brain encoding model is tested with the test set and related processing is performed based on the at least one test set, the step of outputting test metrics utilizing the brain encoding model is performed.

33. One or more non-transitory computer readable media including computer-readable program instructions that, upon execution by one or more computer processors, cause the one or more computer processors to perform operations including: one or more aspects, steps, features, and / or functionality recited in any claim here or set forth elsewhere in the present disclosure.

Images