Augmentative and alternative communication device

Abstract

A method of generating speech from muscular appendage movement thoughts includes receiving and capturing, via an input device, user-generated input signals, which are represented by analog or digital signals, wherein the input device includes at least one electrode array configured to capture neural signals representative of muscular appendage movement thoughts or a manual input device configured to capture physical appendage movements of the user; communicating, by the input device, captured user-generated input signals to the first computing device, the first computing device further including one or more processors executing computer-readable instructions to: generate appendage parameters from the user-generated input signals, wherein the appendage parameters are representative of the muscular appendage movement thoughts or the physical appendage movements; and translate or convert, utilizing an appendage parameters to vocal tract parameters mapping table, the generated appendage parameters to a set of generated vocal tract parameters (which are representative of speech).

Description

Augmentative and Alternative Communication DeviceRELATED APPLICATIONS

[0001] This application claims priority to U.S. provisional patent application serial No. 63 / 452,169, filed March 15, 2023, entitled Augmentative and Alternative Communication Device and U.S. provisional patent application serial No. 63 / 565,522, filed March 14, 2024, entitled Augmentative and Alternative Communication Device, the disclosures of both which are hereby incorporated by reference.BACKGROUND

[0002] This disclosure below relates to an Augmentative and Alternative Communication (AAC) Device.

[0003] Augmentative and Alternative Communication (AAC) devices are used by people who cannot speak or have compromised speech, such as individuals with cerebral palsy (CP), amyotrophic lateral sclerosis (ALS), or individuals who have experienced a stroke. A very rare condition that removes people’s ability to speak is “locked-in syndrome.” This condition results in the affected individual’s loss of control of almost all of their muscles. Typically, the only motor function that remains is that of the eyes and eyelids.

[0004] The book and film “The Diving Bell and the Butterfly” showcase this condition via the story of Jean-Dominique Bauby, who experienced a stroke at age 43 and subsequently experienced locked-in syndrome. A speech therapist assisted him in reestablishing his ability to communicate. Mr. Bauby and his therapist used a physical board with the letters of the alphabet written on it. The sequence of the letters represented the frequency at which the letters occur in the French language. The therapist would point at the letters while reading them out loud in sequence. Mr. Bauby would blink to select a letter, double-blink to end a word, or blink rapidly to start over. Via this method, Mr. Bauby wrote the 131-page book “The Diving Bell and the Butterfly: A Memoir of Life in Death” in 1997. The book was adapted to a film in 2007.

[0005] In 2022, an article with the title “A locked-in man has been able to communicate in sentences by thought alone” was published in the MIT Technology Review. The article describes a patient with ALS who has lost almost all of his motor function. He was reaching the point where using an eye-tracking device combined with a speech-generating device was no longer a viable option. The only remaining alternative within the realm of current technology was a neural implant. The team of researchersimplanted two electrode arrays in the motor cortex of the patient’s brain and tried to decode the data. They asked the patient to imagine muscle movements, which has helped other patients control prosthetic limbs. But, the researchers were unable to make the system work.

[0006] After 12 weeks of no success, the team decided to test a new approach using auditory neurofeedback. The new system provides the patient with real-time information about his brain activity via a continuous tone that was played over a loudspeaker for the patient to hear. The patient’s variation of brain activity in the region in which the electrode arrays are implanted increases or decreases the pitch of the tone.

[0007] Within 2 days, the patient learned to control the pitch of the tone to indicate “yes” with a higher pitched tone and “no” with a lower pitched tone. Next, the research team connected the output signal of the patient’s neural decoder to a form of AAC device the patient had used previously. This newly created system allows the patient to select words and letters, which the devices then speaks out loud, giving the user the ability to communicate full sentences via thought alone.SUMMARY

[0008] Individuals with locked-in syndrome can no longer speak due to a loss of motor function. Current AAC devices do not restore natural speech, do not provide the ability to expressively control intonation, and do not allow the user to sing, hum, or play musical instruments. The subject matter described herein utilizes the output signals of a neural implant that is implanted in the part of an individual’s motor cortex or premotor cortex that typically controls limb movements.

[0009] Output signals of the neural implant that correspond to imagined limb movements are mapped to input parameters of an articulatory speech synthesizer or a musical instrument. The user is then taught to control the articulatory speech synthesizer or musical instrument via imagined limb movements, thereby restoring fully expressive speech. The system can also be used to hum or sing.BRIEF DESCRIPTION OF THE DRAWINGS

[0010] A better understanding of the features, advantages and principles of the present disclosure will be obtained by reference to the following detailed description that sets forth illustrative embodiments, and the accompanying drawings of which:

[0011] FIG. 1 illustrates a screenshot of a software program, developed by Neil Thapen (Institute of Mathematics, Academy of Sciences of the Czech Republic), called“Pink Trombone”, which is an articulatory speech synthesizer that generates human speech via a set of artificial resonance chambers and articulators;

[0012] FIG. 2 shows an image of the Homunculus, created by Encyclopedia Brittanica, illustrating a model of the human motor cortex and somatosensory cortex with mapping locations for each body part that the respective brain tissue is responsible for;

[0013] FIG. 3 shows an electrode array neural implant in a motor cortex, allowing individual finger control of a prosthetic arm according to some implementations;

[0014] FIG. 4 shows a block diagram of an Augmentative and Alternative Communication (AAC) device according to some implementations;

[0015] FIG. 5 illustrates a flowchart of operating an Augmentative and Alternative Communication (AAC) device according to some implementations; and;

[0016] FIG. 6 illustrates a flowchart of playing a synthetic musical instrument according to some implementations.DETAILED DESCRIPTION

[0017] The following detailed description provides a better understanding of the features and advantages of the inventions described in the present disclosure in accordance with the embodiments disclosed herein. Although the detailed description includes many specific embodiments, these are provided by way of example only and should not be construed as limiting the scope of the inventions disclosed herein.

[0018] The subject matter described herein is an Augmentative and Alternative Communication (AAC) device that allows individuals with “locked-in syndrome” to speak, sing, and play musical instruments. This is achieved by implanting at least one electrode array into the motor cortex of an affected individual to extract neural data that correlates with intended muscle movements of the individual.

[0019] The neural data is then sent to a speech synthesizer that replicates a human vocal apparatus and forms a set of artificial articulators. The speech synthesizer or digital articulators output an acoustic speech signal that can be perceived by others as well as the affected individual, establishing a path of auditory neurofeedback. The affected individual thus gains the ability to learn how to control their new artificial articulators to ultimately generate the equivalent of human speech. Alternatively, the speech synthesizer or digital articulators may be used to hum or sing, or it may be replaced with any synthetic musical instrument.

[0020] Existing AAC devices focus on giving the user the ability to select letters, predefined words, picture cards, scenes, or whole sentences, which are then spoken out loud by the device. Current approaches to developing a brain-computer-interface (BCI) for speech production focus on the decoding of complex neural representations of speech and language. This approach has been challenging within current technology and tends to lead to high word error rates, typically on the order of 25% when using a large word corpus. The subject matter described herein takes a different approach and aims to restore motor function, although for a set of artificial or synthetic articulators.

[0021] Human speech is generated by transporting air from the lungs through the larynx, in which the vocal folds are located. The lungs, together with the diaphragm, the intercostal muscles, chest muscles, and neck muscles, as well as the trachea form the respiratory system. The vocal folds and the space between them form the glottis. The vocal folds vibrate within the flow of air, generating sound. The glottis can be viewed as the source of the sound. The vocal folds together with the thyroarytenoid muscles that control the vocal folds as well as several other anatomical structures are known as the phonatory system.

[0022] The sound produced by the phonatory system is then carried through the pharyngeal cavity, the oral cavity, and the nasal cavity, which comprise the resonatory system. Here, the sound is filtered by the shape of these three cavities, resulting in a change of the acoustic characteristics of the sound. The shape of the resonatory system can be adjusted by movement of the articulators, which include the tongue, the lips, the teeth, the upper gum ridge, the soft palate (velum), the hard palate, the uvula, and the pharyngeal wall. Muscles control these articulators, allowing an individual to alter the way their voice sounds, thus creating phonemes, which can be combined to form words and sentences.

[0023] This systematic description of the process of speech production has been named the “source-filter model.” The source of the sound can either be voiced, in which case a Glottal Excitation Model is used to mathematically describe the sound source, or it can be voiceless, in which case a Frictional Excitation Model is used. The sound generated by the source is then filtered by convolving it with a Vocal Tract Transfer Function and subsequently with a Radiation Model, producing the sound as it is radiated from the lips.

[0024] An open-source software program, called “Pink Trombone”, is an articulatory speech synthesizer that simulates the generation of human speech via a set of artificialresonance chambers and articulators. A screenshot of Pink Trombone can be seen in FIG.1. The foundation of this program is a parametric model of the vocal tract area function for vowels and consonants. Other vocal tract simulator implementations of varying levels of complexity are available, such as VTDemo, developed by Mark Huckvale at University College London, United Kingdom, and Vocal TractLab, developed by Peter Birkholz at Technische Universitat Dresden, Germany.

[0025] In the Pink Trombone implementation, a user can manipulate the position and shape of several articulators as well as pitch and breathiness of the voice via their mouse cursor. In this manner, different phonemes can be generated. The system in its original conception has approximately 14 degrees of freedom or input parameters. However, the software implementation appears to have approximately 9 degrees of freedom. To produce meaningful sounds that can be interpreted as human speech, it is possible to reduce the number of degrees of freedom even further. For example, control of pitch and breathiness of the voice is not required for speech comprehension, but merely adds complexity to allow an individual to communicate complex emotions.

[0026] Within the realm of currently available technology, individuals with neural implants have been able to control prosthetic arms with 7 degrees of freedom. Individual finger control has also been achieved via this method. An individual with 4 prosthetic limbs controllable via 4 neural implants would be able to control at least 28 degrees of freedom, significantly more than are required to control a speech synthesizer with artificial articulators. Thus, it may be beneficial to use neural motor / appendage control signals and / or data incorporated or represented therein to control a speech synthesizer with artificial articulators.

[0027] FIG. 2 shows an image of the Homunculus, a model of the human motor cortex and somatosensory cortex with mapping locations for each body part that the respective brain tissue is responsible for.

[0028] The subject matter described herein proposes to control a speech synthesizer with artificial articulators via at least one neural signal from an area of a subject’s brain that is able to generate or receive and understand signals that usually control limbs. In other words, an area of a subject’s brain may be a region of a subject’s brain that is capable of generating or receiving and understanding neural signals representative of appendage parameters. In some embodiments, appendage parameters may include appendage position coordinates, end point coordinates, speed, and / or force information, for example, or a combination thereof. In some implementations, this region or area of thesubject’s brain may be referred to as an appendage controlling area of the brain. In some implementations, the appendage controlling area may be the motor cortex, the pre-motor cortex, or may be referred to as the medial intraparietal (MIP) cortex or the parietal reach region (PRR), for example. In some implementations, neural signals may be analog signal pulses of varying shapes and / or amplitudes.

[0029] The neural signals are decoded to form human-interpretable neural data containing kinematic information, which can then be used to control prosthetic limbs, an artificial human vocal tract, or synthetic musical instruments, for example. Robust functional relationships between neural signals in the human motor cortex and kinematics have been known for several decades. A variety of data analysis and machine learning techniques, such as principal component analysis (PCA), factor analysis (FA), linear dynamical system (LDS) based latent-state estimation, and deep canonical correlation analysis (DCCA) have proven successful in this context and allow neural decoders to generate stable outputs. Decoding models either rely on kinematic states described by neural encoding models or they operate on direct input-output approximations from neural signals to kinematic variables.

[0030] The decoded neural data is then sent to an articulatory speech synthesizer that replicates a human vocal apparatus and forms a set of artificial articulators. The speech synthesizer outputs an acoustic speech signal that can be perceived by others as well as the affected individual, establishing a path of auditory neurofeedback.

[0031] FIG. 3 shows an electrode array neural implant in a motor cortex, allowing individual finger control of a prosthetic arm according to some implementations. As illustrated in FIG. 3, an electrode array neural implant 305 may be inserted into a motor cortex area 310 of a brain 315 of a subject. In FIG. 3, the electrodes may receive and / or capture the neural signals which may be communicated to the motor cortex 310. In FIG. 3, the motor cortex 310 may translate these instructions and / or signals to neural signals that may be transmitted to specific areas of a subject’s body (in this case a subject’s fingers or appendages).

[0032] FIG. 4 shows a block diagram of an Augmentative and Alternative Communication (AAC) device according to some implementations. The AAC system 400 may include a subject’s brain 405, a neural implant 410, neural decoder software 415, at least one computing device 417, a speech synthesizer or digital articulators 420, and / or one or more auditory output devices 425.

[0033] FIG. 5 illustrates a flowchart of operating an Augmentative and Alternative Communication (AAC) device according to some implementations. In some implementations, in step 505, a software program and / or an operator generates a mapping between appendage parameters (“APs”) and vocal tract parameters (“VTPs”). In some implementations, appendage parameters may be movement data (e.g., up, down, left, right, bending an appendage, extending an appendage, rotating an appendage) related to an appendage. In some implementations, appendage parameters may be appendage position coordinates, end point coordinates, speed, and / or force information, for example. In some implementations, vocal tract parameters may include vocal tract shape parameters and voice parameters. In some implementations, vocal tract shape parameters include information about a shape or position of a vocal tract (e.g., articulator positions and tensions). In some implementations, voice parameters may relate to the way sound is generated by the vocal folds (e.g., pitch and breathiness).

[0034] As a representative example of this mapping, in some implementations, a) bending a right index finger (the finger being an appendage) may represent moving a subject’s jaw backwards; b) extending the right index finger may represent moving the jaw forward; c) bending the right middle finger may represent closing the lips; d) extending the right middle finger may represent opening the lips; e) bending the left index finger may represent moving the tongue down; f) extending the left index finger may represent moving the tongue up; g) moving the entire right arm to the right may represent increasing the pitch; h) moving the right arm to the left may represent an increase in volume (e.g., air flow); and i) moving the right arm down may represent decreasing the air flow (whispering). As a further illustration, there are 14 muscular movements (that represent movements of the jaw muscles, the face muscles, the tongue muscles and / or the throat muscles) that alone, or in combination, may constitute a generation of a sound. In some implementations, the mapping of the APs to the VTPs may be configurable and / or customizable. In some implementations, the mapping of the APs to the VTPs may be customized in order to minimize a number of muscular movements necessary to generate at least one sound.

[0035] In some implementations, VTPs may include combinations of muscle group movements or positions rather than controlling individual muscles or isolated muscle groups. The control of such muscle group movements may be achieved by selecting certain VTPs from a range of degrees of freedom. Thus, in some implementations, thesedegrees of freedom may represent characteristics of speech production, such as place, voicing, or manner of articulation.

[0036] In this context, place refers to the position within the human vocal tract at which a sound is produced. For example, phonemes, such as consonants, may be arranged along an axis, representing the place of articulation in a human vocal tract. A vocal tract parameter may select a place of articulation, which then refers to a specific phoneme. For example, phonemes, such as / p / , / b / , / m / , and / w / are placed at the front of the mouth, involving both lips. Phonemes, such as / t / , / d / , and / s / are placed slightly further back in the mouth, involving the tongue and the alveolar ridge. Phonemes, such as / k / , / g / , and / h / are placed in the back of the mouth.

[0037] In a similar fashion, in some implementations, phonemes, such as vowels, may be arranged along two intersecting axes. A first axis may represent the horizontal position of the place of articulation, such as front, central, or back. In some implementations, a second axis may represent the vertical position of the place of articulation, such as high, mid, or low. Therefore, it may be possible to select the place of articulation of any phoneme with two degrees of freedom.

[0038] Within human speech, some phonemes are voiced, while others are voiceless. Voicing refers to whether the vocal folds are vibrating during the production of the phoneme. For example, / z / or / b / are voiced, while / s / and / p / are voiceless. Voicing is a binary selector. Therefore, voicing may be selected with one degree of freedom.

[0039] Manner of articulation refers to how a phoneme is produced. For example, phonemes such as / b / , / p / , / t / , / d / , / k / , and / g / are considered “stops.” These phonemes are produced by stopping all airflow and then rapidly increasing airflow upon production of the phoneme. Similarly, some phonemes are considered affricates, fricatives, liquids, glides, and nasals. These types of phonemes are discrete items that may be placed along an axis. It may also be possible to combine voicing selection and manner selection to one degree of freedom.

[0040] In this manner, in some implementations, it may be possible to reduce the entirety of human speech production to 5 degrees of freedom, for example (1) and (2) place of articulation, (3) manner and voicing, (4) pitch, and (5) breathiness or volume level.

[0041] In some implementations, in step 510, the generated AP to VTP mapping table may be stored in one or more memory devices of a first computing device. In some implementations, the first computing device may be a desktop computing device, a laptopcomputing device, a tablet computing device, a smartphone, and / or an embedded device that receives signals from the neural implant and transmits representative vocal tract parameters to a speech synthesizer. In some implementations, the generated AP to VTP mapping table may be stored in one or more memory devices of the speech synthesizer 420.

[0042] In some implementations, a user or subject may need to train themselves on the appendage movements that are necessary to create sounds and / or speech. In other words, the user or subject may need to train themselves on the AP to VTP mapping. Please note that in some implementations, the user or subject may not actually move any appendages. The user or subject may just generate neural signals that represent moving the appendages. Neural signals for muscle movements may be communicated and / or transmitted to an appendage controlling area of a subject’s or user’s brain. In some implementations, the appendage controlling area may be a motor cortex. In some implementations, the appendage controlling area may be a pre-motor cortex, a medial intraparietal (MIP) cortex, or a parietal reach region (PRR), for example.

[0043] In some implementations, a neural implant 410 may be implanted into a motor cortex, a pre-motor cortex, an MIP cortex, or a parietal reach region, for example, of a subject’s brain 405. In some implementations, a neural implant 410 may be a plurality of electrodes or an electrode array.

[0044] In some implementations, in step 515, a neural electrode array may capture neural signals, where the neural signals are analog electrical pulses of varying shapes and amplitudes, representing intended movements of appendages. In some implementations, in step 517, the neural electrode array may communicate the captured neural signals to a first computing device 417, which may include neural decoder software 415. In some implementations, in step 520, the neural decoder software 415 in the first computing device 417 may generate appendage parameters based at least in part on the received neural signals representing intended movements of appendages.

[0045] In some implementations, in step 522, the neural decoder software 415 in the first computing device 417 may utilize the AP to VTP mapping in the database to convert the appendage parameters to vocal tract parameters. In some implementations, in step 525, the neural decoder software 415 in the first computing device 417 may communicate the set of generated vocal tract parameters to a speech synthesizer or digital articulators 420.

[0046] In some implementations, in step 530, the speech synthesizer or digital articulators 420 may output an acoustic speech signal corresponding to and / or based at least in part on the generated vocal tract parameters.

[0047] In some implementations, in step 535, the speech synthesizer or digital articulators 420 may communicate the acoustic speech signal to at least one auditory output device 425 for perception by the subject and / or other organisms or users. In some implementations, the at least one auditory output device 425 may include at least one earphone, at least one headphone, a tactile transducer, or a hearing aid output transducer. In some implementations, the at least one auditory output device may be a cochlear implant electrode, an auditory brainstem implant electrode, a neural implant electrode, a transcranial electrical stimulation device or a transcranial magnetic stimulation (TMS) device.

[0048] In an alternative embodiment, in step 540, the neural decoder software 415 in the first computing device 417 may communicate the generated vocal tract parameters to a second or another computing device 418. In these embodiments, in step 545, the second or another computing device 418 may communicate the generated vocal tract parameters to the speech synthesizer or digital articulators 420. Then, the speech synthesizer or digital articulators may generate an acoustic speech signal, as represented by step 530, and may communicate the acoustic speech signal to at least one auditory output device 425 for perception by the subject or other organisms or users, as represented by step 535.

[0049] The subject matter described herein may be modified to control synthetic musical instruments instead of artificial articulators. For example, a user of the subject matter system and method described herein may play an instrument, such as a trombone, a trumpet, or a theremin via thought regarding appendage movements and / or positions, which are then translated to positions or movements within the degrees of freedom of the instrument, ultimately emitting sound from the synthetic musical instrument.

[0050] FIG. 6 illustrates a flowchart of playing a synthetic musical instrument according to some implementations. In some implementations, in step 605, a software program and / or an operator generates a mapping between appendage parameters (“APs”) and musical instrument parameters (“MIPs”). In some implementations, for example, when playing a synthetic trombone, musical instrument parameters may include slide position (which adjusts the tubing length and alters the pitch), embouchure (which describes lip tension and shape, controlling timbre and pitch), air pressure and flow (which influence volume and pitch stability), and tonguing (which affects attack,articulation, and tone). In some implementations, for example, when playing a synthetic theremin, musical instrument parameters may include pitch and volume. Other synthetic musical instruments may have representative musical instrument parameters.

[0051] In some implementations, in step 610, the generated AP to MIP mapping table may be stored in one or more memory devices of a first computing device. In some implementations, in step 615, a neural electrode array may capture neural signals, where the neural signals are analog electrical pulses of varying shapes and amplitudes, representing intended movements of appendages. In some implementations, in step 617, the neural electrode array may communicate the captured neural signals to a first computing device 417, which may include neural decoder software 415. In some implementations, in step 620, the neural decoder software 415 in the first computing device 417 may generate appendage parameters based at least in part on the received neural signals representing intended movements of appendages. In some implementations, in step 622, the neural decoder software 415 may utilize the AP to MIP mapping table to convert the appendage parameters to musical instrument parameters. The neural decoder software 415, as described previously, may include computer- readable instructions stored in a memory that are executable by one or more processors of a computing device to perform functions or operations described below.

[0052] In some implementations, in step 625, the neural decoder software 415 may communicate the generated musical instrument parameters to a synthetic musical instrument. In some implementations, in step 630, the synthetic musical instrument may output an acoustic audio signal corresponding to and / or based at least in part on the generated musical instrument parameters. In some implementations, in step 635, the synthetic musical instrument may communicate the acoustic audio signal to at least one auditory output device 425 for perception by the subject and / or other organisms or users.

[0053] In an alternative embodiment, in step 640, the neural decoder software 415 in the first computing device 417 may communicate the generated musical instrument parameters to a second or another computing device 418. In these embodiments, in step 645, the second or another computing device 418 may communicate the generated musical instrument parameters to the synthetic musical instrument. Then, the synthetic musical instrument may generate an acoustic audio signal, as represented by step 630, and may communicate the acoustic audio signal to at least one auditory output device 425 for perception by the subject or other organisms or users, as represented by step 635.

[0054] The claimed subject matter may also be implemented for testing purposes, for example, with individuals whose speech is intact and who control prosthetic limbs via neural implants. The prosthetic limbs may be disconnected and instead, the output data that controls each degree of freedom of each prosthetic limb may be fed into a neural decoder software 415 and mapped via the AP to VTP mapping table to the input parameters of a speech synthesizer or digital articulators 420 or a synthetic musical instrument.

[0055] The claimed subject matter process may require the help of a speech therapist in order to determine sounds that may be represented by at least one appendage movement (and appendage parameters (APs)). In some implementations, a training procedure or process with a speech therapist may be modeled after speech acquisition in healthy children. For example, the training program may mimic the following stages: (1) reflexive vocalization, (2) cooing, (3) vocal play, (4) canonical babbling, (5) jargon, (6) holophrases. A holophrase is defined as a single word expressing a complex idea. Once the individual has reached the holophrase stage, speech utilizing the claimed subject matter system and / or method may be mostly fluent, given that individuals for whom the claimed subject matter system and method may be of interest typically have fully developed language skills.

[0056] For a child, the entire training process described immediately above takes approximately 18 months. For an individual whose speech was previously fully developed, this training process described immediately above may be significantly shorter. However, because individuals and / or subjects may require different levels of assistance and / or guidance, the training process described immediately above may also be significantly longer. In some implementations, an individual’s ability to learn may be compromised due to age or other neurological conditions, such as Alzheimer’s disease or Parkinson’s disease, for example, or from any trauma to the brain, including traumatic brain injury or stroke.

[0057] In some implementations, it may also be possible to customize the training process to the individual. In speech therapy, this may be referred to as “client-specific targets,” which do not necessarily have to follow a child’s developmental language stages. An adult who can comprehend the position and function of each articulator, for example, may be able to acquire speech utilizing the claimed subject matter system and method much faster.

[0058] Current AAC devices allow individuals to select letters or words that the AAC device then speaks out loud, which is a long, difficult, and tiresome process. The novel aspect of the claimed subject matter is that it allows individuals to utilize appendage movements and / or positions to produce customized phonemes and thus communicate in a more natural fashion. In other words, the claimed subject matter herein allows an individual to form any combination of phonemes, (which may be customizable,) thus regaining an ability to use language productively, meaning that novel words can be created rather than being restricted to a predefined vocabulary.

[0059] A clear benefit of the claimed subject matter is that current technology does not allow individuals to produce phonemes (or customizable phonemes) via thought alone in contrast to the described subject matter. In addition, current technology does not utilize neurofeedback, and specifically current technology does not use neural data from a part of the motor cortex or pre-motor cortex that controls limbs or appendages to control artificial articulators or speech synthesizers; which are clearly different body parts than what the neural data was originally intended for. In contrast, the claimed subject matter does utilize neural data from a part of the motor cortex or pre-motor cortex. Further, current technology does not leverage neural plasticity to map neural data intended for a certain body part to be mapped to a different body part, which is in contrast to the claimed subject matter. Finally, current technology does not provide training procedures that mimic speech acquisition of a healthy child or speech therapy with client-specific targets, which is in contrast to the claimed subject matter.

[0060] Another problem with currently available AAC device technology is that the technology is not future-proof. As an example, if the manufacturer of a current AAC device were to go out of business or discontinue support for the AAC device, the user would be restricted to a current vocabulary or functionality of the device without assistance or improvements. This may be problematic because, as language evolves, the user may not be able to communicate novel words independently on their own. Because the claimed subject matter is modifiable and / or configurable, the device may not restrict the user in this way, since it is dictionary-independent (e.g., any phoneme or combination of phonemes may be created with any given AP to VTP mapping table). Further, the AP to VTP mapping table may be updated.

[0061] Additional benefits of the claimed AAC device may include, but are not limited to: 1) the claimed subject matter does not restrict individuals to a predefined vocabulary and thus may allow them to use language productively, forming new words with novelmeanings via associated appendage movement or position parameters and the claimed subject matter is dictionary-independent and may be used for any language.

[0062] Additional uses or applications of the claimed AAC device may include, but are not limited to: 1) The claimed subject matter may be combined with speech-to-text algorithms to create a thought-to-text mechanism that may allow a user to control a computer and / or interact with voice assistants, such as Apple’s Siri and Amazon’s Alexa; for example, in step 535, instead of communicating the acoustic speech signal to at least one auditory output device 425 for perception by the subject or other organism, the acoustic speech signal may be fed directly into the audio input of a voice assistant; 2) The claimed subject matter may be used to dictate voice memos via thought; 3) The claimed subject matter may be used to play musical instruments; and 4) The claimed subject matter may be used in combination with delayed auditory feedback to assist individuals who stutter become fluent.

[0063] Additional uses or applications of the claimed AAC device may also include the following: 5) The claimed subject matter may be used for accent coaching and to learn new languages; 6) The claimed subject matter may assist users in overcoming phonological deafness, a phenomenon that describes how children lose the ability to detect differences in phonemes that are not part of their native language around 10 to 12 months of age; and 7) The claimed subject matter may be used in combination with wireless audio transmission methods, such as Bluetooth Auracast, for example, to transmit the generated speech signals to another individual who may then listen to the speech signals with earphones, headphones, hearing aids, cochlear implants, or auditory brainstem implants that have the ability to decode the transmitted signals; for example, in step 540, the first computing device 417 may be a device used by a first person to generate a speech signal, while the second computing device 418 may be a device used by a second person to receive speech signals and to convert them into auditory signals, which the second person’s brain knows how to interpret.

[0064] Additional uses or applications of the claimed AAC device may also include the following: 8) The claimed subject matter may be used as a means of establishing direct brain-to-brain communication if the recipient has a brain implant that allows decoding of the speech signals and injecting the generated audio signals into the auditory cortex of the recipient; for example, in step 540, the first computing device 417 may be a device used by a first person to convert thoughts into a speech signal, while the second computing device 418 may be a device used by a second person to receive speech-basedthoughts and convert them into auditory signals, which the second person’s brain knows how to interpret, thereby effectively establishing a pathway of telepathy; and 9) The claimed subject matter may be used to communicate faster than is currently possible via human speech. The rate of human speech is limited by the physical movements of the articulators at approximately 125 word per minute. However, with synthetic articulators, this limitation may no longer exist. Thus, with advances in accuracy of decoding neural data, it may be possible to speed up synthetic speech in the same way a screen reader does for blind people. As we know from blind people, it is possible for the human brain to learn to interpret acoustic signals with such rapid speech up to a rate of approximately 400 words per minute. Ultimately, the claimed subject matter may establish a direct brain-to-brain communication channel that vastly outpaces the data rate of biological human speech and may allow communication at the speed of thought; and the claimed subject matter may enable animals, such as non-human primates or cats, for example, to speak and be understood (e.g., Dr. Doolittle could understand speech of animals) if put through a training program with customized appendage positional and movement parameters paired up with specific sounds, words, and / or phrases.

[0065] In some embodiments, a method of generating speech from muscular appendage movement thoughts includes generating an appendage parameters (AP) to vocal tract parameters (VTP) mapping table; storing, in one or more memory devices of a first computing device 417 the AP to VTP mapping table; receiving and capturing, via at least one electrode array 410 installed or resident in a region of subject’s brain 405, analog signals, including electrical pulses of varying shapes and amplitudes, representative of muscular appendage movement thoughts, the region of subject’s brain 405 being capable of generating or receiving and understanding the analog signals representative of muscular appendage movement thoughts; communicating, by the at least one electrode array 410, captured analog signals to the first computing device 417, the first computing device 417 further including one or more processors, the one or more processors executing computer-readable instructions to: generate appendage parameters from the captured analog signals, wherein the appendage parameters are representative of muscular appendage movement thoughts; translate or convert, utilizing the AP to VTP mapping table, the generated appendage parameters to a set of generated vocal tract parameters, wherein the set of generated vocal tract parameters is representative of speech; communicating the set of generated vocal tract parameters to a speech synthesizer or digital articulators 420; outputting, via the speech synthesizer or digital articulators420, an acoustic speech signal corresponding to the set of generated vocal tract parameters; and communicating the acoustic speech signal to at least one auditory output device 425 for audio perception by the subject or other organisms.

[0066] In some embodiments, the at least one auditory output device 425 may include at least one earphone, at least one headphone, a tactile transducer, a hearing aid output transducer, a cochlear implant electrode, a brain implant electrode, a transcranial electrical stimulation device, or a transcranial magnetic stimulation device. In some embodiments, the appendage parameters may include appendage position coordinates, end point coordinates, speed parameters, or force parameters.

[0067] In some alternative embodiments, the first computing device 417 may communicate the set of generated vocal tract parameters to a second computing device 418. The second computing device 418 may communicate the set of generated vocal tract parameters to a speech synthesizer or digital articulators 420. The speech synthesizer or digital articulators 420 may output an acoustic speech signal corresponding to the set of generated vocal tract parameters to at least one auditory output device 425 for audio perception by the subject or other organisms.

[0068] In some embodiments, an augmentative and alternative communication system includes a first computing device 417; at least one processor; at least one memory device; computer-readable instructions, the computer-readable instructions executable by the at least one processor to: receive an appendage parameters (AP) to vocal tract parameters (VTP) mapping table; store, in the at least one memory device of a first computing device 417 the AP to VTP mapping table; receive captured analog signals representative of muscular appendage movement thoughts, at the first computing device 417, the captured analog signals being received from at least one electrode array 410 installed or resident in a region of a subject’s brain 405, the region of the subject’s brain 405 being capable of generating or receiving and understanding analog signals representative of the muscular appendage movement thoughts; generate appendage parameters from the captured analog signals, wherein the appendage parameters are representative of muscular appendage movement thoughts; and translate or convert, utilizing the AP to VTP mapping table stored in the at least one memory device, the generated appendage parameters to a set of generated vocal tract parameters, wherein the set of generated vocal tract parameters is representative of speech. In some embodiments, the system may further include a speech synthesizer or digital articulators 420, where the first computing device 417 communicates the set of generated vocal tract parameters to the speech synthesizer ordigital articulators 420 and the speech synthesizer or digital articulators 420 output an acoustic speech signal corresponding to the set of generated vocal tract parameters to at least one auditory output device 425 for audio perception by a subject or other organisms.

[0069] In alternative embodiments, the first computing device 417 may communicate the set of generated vocal tract parameters to a second computing device 418 and the second computing device 418 may communicate the set of generated vocal tract parameters to the speech synthesizer or digital articulators 420. In these embodiments, the speech synthesizer or digital articulators 420 may output the set of generated vocal tract parameters to at least one auditory output device 425 for audio perception by a subject or other organisms.

[0070] In some embodiments, a method of generating speech from muscular appendage movement thoughts, including generating an appendage parameters (AP) to vocal tract parameters (VTP) mapping table, wherein the appendage parameters are derived from user interactions with an input device; storing, in one or more memory devices of a first computing device 417 the AP to VTP mapping table; receiving and capturing, via the input device, user-generated input signals, which are represented by analog or digital signals, including electrical pulses of varying shapes and amplitudes, wherein the input device includes at least one electrode array 410 configured to capture neural signals representative of muscular appendage movement thoughts or a manual input device configured to capture physical appendage movements of the user; communicating, by the input device, captured user-generated input signals to the first computing device 417, the first computing device 417 further including one or more processors, the one or more processors executing computer-readable instructions to: generate appendage parameters from the user-generated input signals, wherein the appendage parameters are representative of the muscular appendage movement thoughts or the physical appendage movements; translate or convert, utilizing the AP to VTP mapping table, the generated appendage parameters to a set of generated vocal tract parameters, wherein the set of generated vocal tract parameters is representative of speech; and communicate, by the first computing device 417, the set of generated vocal tract parameters to a speech synthesizer or digital articulators 420. In some embodiments, the speech synthesizer or digital articulators 420 may output an acoustic speech signal corresponding to the set of generated vocal tract parameters to at least one auditory output device 425 for audio perception by the subject or other organisms.

[0071] In some embodiments, an augmentative and alternative communication system, may include a first computing device 417; at least one processor; at least one memory device; computer-readable instructions, the computer-readable instructions executable by the at least one processor to: receive an appendage parameters (AP) to vocal tract parameters (VTP) mapping table; store, in the at least one memory device of a first computing device 417 the AP to VTP mapping table; receive and capture, via the input device, user-generated input signals, which are represented by analog or digital signals, including electrical pulses of varying shapes and amplitudes, wherein the input device includes at least one electrode array 410 configured to capture neural signals representative of muscular appendage movement thoughts or a manual input device configured to capture physical appendage movements of the user; generate appendage parameters from the captured analog signals, wherein the appendage parameters are representative of the muscular appendage movement thoughts or the physical appendage movements; translate or convert, utilizing the AP to VTP mapping table stored in the at least one memory device, the generated appendage parameters to a set of generated vocal tract parameters, wherein the set of generated vocal tract parameters is representative of speech; and communicate the set of generated vocal tract parameters to the speech synthesizer or digital articulators 420. In some embodiments, the speech synthesizer or digital articulators 420 may output an acoustic speech signal corresponding to the set of generated vocal tract parameters to at least one auditory output device 425 for audio perception by a subject or other organisms.

[0072] A method of generating sound from muscular appendage movement thoughts may include generating an appendage parameters (AP) to musical instrument parameters (MIP) mapping table; storing, in one or more memory devices of a first computing device 417 the AP to MIP mapping table; receiving and capturing, via at least one electrode array 410 installed or resident in a region of subject’s brain 405, analog signals, including electrical pulses of varying shapes and amplitudes, representative of muscular appendage movement thoughts, the region of subject’s brain 405 being capable of generating or receiving and understanding the analog signals representative of muscular appendage movement thoughts; communicating, by the at least one electrode array 410, captured analog signals to the first computing device 417, the first computing device 417 further including one or more processors, the one or more processors executing computer- readable instructions to: generate appendage parameters from the captured analog signals, wherein the appendage parameters are representative of muscular appendage movementthoughts; translate or convert, utilizing the AP to MIP mapping table, the generated appendage parameters to a set of generated musical instrument parameters, wherein the set of generated musical instrument parameters is representative of sound.

[0073] In some embodiments, an augmentative and alternative communication system may include a first computing device 417; at least one processor; at least one memory device; computer-readable instructions, the computer-readable instructions executable by the at least one processor to: receive an appendage parameters (AP) to musical instrument parameters (MIP) mapping table; store, in the at least one memory device of a first computing device 417 the AP to MIP mapping table; receive captured analog signals representative of muscular appendage movement thoughts, at the first computing device 417, the captured analog signals being received from at least one electrode array 410 installed or resident in a region of a subject’s brain 405, the region of the subject’s brain 405 being capable of generating or receiving and understanding analog signals representative of the muscular appendage movement thoughts; generate appendage parameters from the captured analog signals, wherein the appendage parameters are representative of muscular appendage movement thoughts; and translate or convert, utilizing the AP to MIP mapping table stored in the at least one memory device, the generated appendage parameters to a set of generated musical instrument parameters, wherein the set of generated musical instrument parameters is representative of sound. As detailed above, the computing devices and systems described and / or illustrated herein broadly represent any type or form of computing device or system capable of executing computer-readable instructions, such as those contained within the modules described herein. In their most basic configuration, these computing device(s) may each comprise at least one memory device and at least one physical processor.

[0074] The term “memory” or “memory device,” as used herein, generally represents any type or form of volatile or non-volatile storage device or medium capable of storing data and / or computer-readable instructions. In one example, a memory device may store, load, and / or maintain one or more of the modules described herein. Examples of memory devices comprise, without limitation, Random Access Memory (RAM), Read Only Memory (ROM), flash memory, Hard Disk Drives (HDDs), Solid-State Drives (SSDs), optical disk drives, caches, variations or combinations of one or more of the same, or any other suitable storage memory.

[0075] In addition, the term “processor” or “physical processor,” as used herein, generally refers to any type or form of hardware-implemented processing unit capable ofinterpreting and / or executing computer-readable instructions. In one example, a physical processor may access and / or modify one or more modules stored in the above-described memory device. Examples of physical processors comprise, without limitation, microprocessors, microcontrollers, Central Processing Units (CPUs), Field- Programmable Gate Arrays (FPGAs) that implement softcore processors, Application- Specific Integrated Circuits (ASICs), portions of one or more of the same, variations or combinations of one or more of the same, or any other suitable physical processor.

[0076] Although illustrated as separate elements, the method steps described and / or illustrated herein may represent portions of a single application. In addition, in some embodiments one or more of these steps may represent or correspond to one or more software applications or programs that, when executed by a computing device, may cause the computing device to perform one or more tasks, such as the method step.

[0077] In addition, one or more of the devices described herein may transform data, physical devices, and / or representations of physical devices from one form to another. For example, one or more of the devices recited herein may receive image data of a sample to be transformed, transform the image data, output a result of the transformation to determine a 3D process, use the result of the transformation to perform the 3D process, and store the result of the transformation to produce an output image of the sample. Additionally, or alternatively, one or more of the modules recited herein may transform a processor, volatile memory, non-volatile memory, and / or any other portion of a physical computing device from one form of computing device to another form of computing device by executing on the computing device, storing data on the computing device, and / or otherwise interacting with the computing device.

[0078] The term “computer-readable medium,” as used herein, generally refers to any form of device, carrier, or medium capable of storing or carrying computer-readable instructions. Examples of computer-readable media comprise, without limitation, transmission-type media, such as carrier waves, and non-transitory-type media, such as magnetic-storage media (e.g., hard disk drives, tape drives, and floppy disks), optical- storage media (e.g., Compact Disks (CDs), Digital Video Disks (DVDs), and BLU-RAY disks), electronic-storage media (e.g., solid-state drives and flash media), and other distribution systems.

[0079] A person of ordinary skill in the art will recognize that any process or method disclosed herein can be modified in many ways. The process parameters and sequence of the steps described and / or illustrated herein are given by way of example only and can bevaried as desired. For example, while the steps illustrated and / or described herein may be shown or discussed in a particular order, these steps do not necessarily need to be performed in the order illustrated or discussed.

[0080] The various exemplary methods described and / or illustrated herein may also omit one or more of the steps described or illustrated herein or comprise additional steps in addition to those disclosed. Further, a step of any method as disclosed herein can be combined with any one or more steps of any other method as disclosed herein.

[0081] Unless otherwise noted, the terms “connected to” and “coupled to” (and their derivatives), as used in the specification and claims, are to be construed as permitting both direct and indirect (i.e., via other elements or components) connection. In addition, the terms “a” or “an,” as used in the specification and claims, are to be construed as meaning “at least one of.” Finally, for ease of use, the terms “including” and “having” (and their derivatives), as used in the specification and claims, are interchangeable with and shall have the same meaning as the word “comprising.”

[0082] The processor as disclosed herein can be configured with instructions to perform any one or more steps of any method as disclosed herein.

[0083] As used herein, the term “or” is used inclusively to refer items in the alternative and in combination.

[0084] As used herein, characters such as numerals refer to like elements.

[0085] Embodiments of the present disclosure have been shown and described as set forth herein and are provided by way of example only. One of ordinary skill in the art will recognize numerous adaptations, changes, variations and substitutions without departing from the scope of the present disclosure. Several alternatives and combinations of the embodiments disclosed herein may be utilized without departing from the scope of the present disclosure and the inventions disclosed herein. Therefore, the scope of the presently disclosed inventions shall be defined solely by the scope of the appended claims and the equivalents thereof.

Claims

CLAIMSWHAT IS CLAIMED IS:

1. A method of generating speech from muscular appendage movement thoughts, comprising: generating an appendage parameters (AP) to vocal tract parameters (VTP) mapping table; storing, in one or more memory devices of a first computing device the AP to VTP mapping table; receiving and capturing, via at least one electrode array installed or resident in a region of subject’s brain, analog signals, including electrical pulses of varying shapes and amplitudes, representative of muscular appendage movement thoughts, the region of subject’s brain being capable of generating or receiving and understanding the analog signals representative of muscular appendage movement thoughts; communicating, by the at least one electrode array, captured analog signals to the first computing device, the first computing device further including one or more processors, the one or more processors executing computer-readable instructions to: generate appendage parameters from the captured analog signals, wherein the appendage parameters are representative of muscular appendage movement thoughts; and translate or convert, utilizing the AP to VTP mapping table, the generated appendage parameters to a set of generated vocal tract parameters, wherein the set of generated vocal tract parameters is representative of speech.

2. The method of claim 1, further comprising the first computing device communicating the set of generated vocal tract parameters to a speech synthesizer or digital articulators.

3. The method of claim 2, further comprising: outputting, via the speech synthesizer or digital articulators, an acoustic speech signal corresponding to the set of generated vocal tract parameters.

4. The method of claim 3, further comprising: communicating the acoustic speech signal to at least one auditory output device for audio perception by the subject or other organisms.

5. The method of claim 4, wherein the at least one auditory output device includes at least one earphone, at least one headphone, a tactile transducer, a hearing aid output transducer, a cochlear implant electrode, a brain implant electrode, a transcranial electrical stimulation device, or a transcranial magnetic stimulation device.

6. The method of claim 1, wherein the appendage parameters include appendage position coordinates, end point coordinates, speed parameters, or force parameters.

7. The method of claim 1, further comprising the first computing device communicating the set of generated vocal tract parameters to a second computing device.

8. The method of claim 7, further comprising the second computing device communicating the set of generated vocal tract parameters to a speech synthesizer or digital articulators.

9. The method of claim 8, outputting, via the speech synthesizer or digital articulators, an acoustic speech signal corresponding to the set of generated vocal tract parameters.

10. The method of claim 9, further comprising: communicating the acoustic speech signal to at least one auditory output device for audio perception by the subject or other organisms.

11. The method of claim 10, wherein the at least one auditory output device includes at least one earphone, at least one headphone, a tactile transducer, a hearing aid output transducer, a cochlear implant electrode, a brain implant electrode, a transcranial electrical stimulation device, or a transcranial magnetic stimulation device.

12. An augmentative and alternative communication system, comprising: a first computing device; at least one processor; at least one memory device; computer-readable instructions, the computer-readable instructions executable by the at least one processor to: receive an appendage parameters (AP) to vocal tract parameters (VTP) mapping table; store, in the at least one memory device of the first computing device the AP to VTP mapping table; receive captured analog signals representative of muscular appendage movement thoughts, at the first computing device, the captured analog signals being received from at least one electrode array installed or resident in a region of a subject’s brain, the region of the subject’s brain being capable of generating or receiving and understanding analog signals representative of the muscular appendage movement thoughts; generate appendage parameters from the captured analog signals, wherein the appendage parameters are representative of muscular appendage movement thoughts; andtranslate or convert, utilizing the AP to VTP mapping table stored in the at least one memory device, the generated appendage parameters to a set of generated vocal tract parameters, wherein the set of generated vocal tract parameters is representative of speech.

13. The system of claim 12, further comprising a speech synthesizer or digital articulators, the first computing device communicating the set of generated vocal tract parameters to the speech synthesizer or digital articulators.

14. The system of claim 13, further comprising: outputting, via the speech synthesizer or digital articulators, an acoustic speech signal corresponding to the set of generated vocal tract parameters.

15. The system of claim 14, further comprising at least one auditory output device, the speech synthesizer or digital articulators communicating the acoustic speech signal to the at least one auditory output device for audio perception by a subject or other organisms.

16. The system of claim 15, wherein the at least one auditory output device includes at least one earphone, at least one headphone, a tactile transducer, a hearing aid output transducer, a cochlear implant electrode, an auditory brainstem implant electrode, a neural implant electrode, a transcranial electrical stimulation device, or a transcranial magnetic stimulation device.

17. The system of claim 12, wherein the appendage parameters include appendage position coordinates, end point coordinates, speed parameters, or force parameters.

18. The system of claim 12, further comprising a second computing device, the first computing device communicating the set of generated vocal tract parameters to the second computing device.

19. The system of claim 18, further comprising a speech synthesizer or digital articulators, the second computing device communicating the set of generated vocal tract parameters to the speech synthesizer or digital articulators.

20. The system of claim 19, outputting, via the speech synthesizer or digital articulators, an acoustic speech signal corresponding to the set of generated vocal tract parameters.

21. The system of claim 20, further comprising at least one auditory output device, the speech synthesizer or digital articulators communicating the acoustic speechsignal to the at least one auditory output device for audio perception by a subject or other organisms.

22. The system of claim 21, wherein the at least one auditory output device includes at least one earphone, at least one headphone, a tactile transducer, a hearing aid output transducer, a cochlear implant electrode, an auditory brainstem implant electrode, a neural implant electrode, a transcranial electrical stimulation device, or a transcranial magnetic stimulation device.

23. A method of generating speech from muscular appendage movement thoughts, comprising: generating an appendage parameters (AP) to vocal tract parameters (VTP) mapping table, wherein the appendage parameters are derived from user interactions with an input device; storing, in one or more memory devices of a first computing device the AP to VTP mapping table; receiving and capturing, via the input device, user-generated input signals, which are represented by analog or digital signals, including electrical pulses of varying shapes and amplitudes, wherein the input device includes at least one electrode array configured to capture neural signals representative of muscular appendage movement thoughts or a manual input device configured to capture physical appendage movements of the user; communicating, by the input device, captured user-generated input signals to the first computing device, the first computing device further including one or more processors, the one or more processors executing computer-readable instructions to: generate appendage parameters from the user-generated input signals, wherein the appendage parameters are representative of the muscular appendage movement thoughts or the physical appendage movements; and translate or convert, utilizing the AP to VTP mapping table, the generated appendage parameters to a set of generated vocal tract parameters, wherein the set of generated vocal tract parameters is representative of speech.

24. The method of claim 23, further comprising the first computing device communicating the set of generated vocal tract parameters to a speech synthesizer or digital articulators.

25. The method of claim 24, further comprising: outputting, via the speech synthesizer or digital articulators, an acoustic speech signal corresponding to the set of generated vocal tract parameters.

26. The method of claim 25, further comprising: communicating the acoustic speech signal to at least one auditory output device for audio perception by the subject or other organisms.

27. The method of claim 26, wherein the at least one auditory output device includes at least one earphone, at least one headphone, a tactile transducer, a hearing aid output transducer, a cochlear implant electrode, an auditory brainstem implant electrode, a neural implant electrode, a transcranial electrical stimulation device, or a transcranial magnetic stimulation device.

28. The method of claim 23, wherein the appendage parameters include appendage position coordinates, end point coordinates, speed parameters, or force parameters.

29. The method of claim 23, further comprising the first computing device communicating the set of generated vocal tract parameters to a second computing device.

30. The method of claim 29, further comprising the second computing device communicating the set of generated vocal tract parameters to a speech synthesizer or digital articulators.

31. The method of claim 30, outputting, via the speech synthesizer or digital articulators, an acoustic speech signal corresponding to the set of generated vocal tract parameters.

32. The method of claim 31, further comprising: communicating the acoustic speech signal to at least one auditory output device for audio perception by the subject or other organisms.

33. The method of claim 32, wherein the at least one auditory output device includes at least one earphone, at least one headphone, a tactile transducer, a hearing aid output transducer, a cochlear implant electrode, an auditory brainstem implant electrode, a neural implant electrode, a transcranial electrical stimulation device, or a transcranial magnetic stimulation device.

34. An augmentative and alternative communication system, comprising: a first computing device; at least one processor; at least one memory device; computer-readable instructions, the computer-readable instructions executable by the at least one processor to: receive an appendage parameters (AP) to vocal tract parameters (VTP) mapping table;store, in the at least one memory device of a first computing device the AP to VTP mapping table; receive and capture, via the input device, user-generated input signals, which are represented by analog or digital signals, including electrical pulses of varying shapes and amplitudes, wherein the input device includes at least one electrode array configured to capture neural signals representative of muscular appendage movement thoughts or a manual input device configured to capture physical appendage movements of the user; generate appendage parameters from the captured analog signals, wherein the appendage parameters are representative of the muscular appendage movement thoughts or the physical appendage movements; and translate or convert, utilizing the AP to VTP mapping table stored in the at least one memory device, the generated appendage parameters to a set of generated vocal tract parameters, wherein the set of generated vocal tract parameters is representative of speech.

35. The system of claim 34, further comprising a speech synthesizer or digital articulators, the first computing device communicating the set of generated vocal tract parameters to the speech synthesizer or digital articulators.

36. The system of claim 35, further comprising: outputting, via the speech synthesizer or digital articulators, an acoustic speech signal corresponding to the set of generated vocal tract parameters.

37. The system of claim 36, further comprising at least one auditory output device, the speech synthesizer or digital articulators communicating the acoustic speech signal to the at least one auditory output device for audio perception by a subject or other organisms.

38. The system of claim 37, wherein the at least one auditory output device includes at least one earphone, at least one headphone, a tactile transducer, a hearing aid output transducer, a cochlear implant electrode, an auditory brainstem implant electrode, a neural implant electrode, a transcranial electrical stimulation device, or a transcranial magnetic stimulation device.

39. The system of claim 34, wherein the appendage parameters include appendage position coordinates, end point coordinates, speed parameters, or force parameters.

40. The system of claim 34, further comprising a second computing device, the first computing device communicating the set of generated vocal tract parameters to the second computing device.

41. The system of claim 40, further comprising a speech synthesizer or digital articulators, the second computing device communicating the set of generated vocal tract parameters to the speech synthesizer or digital articulators.

42. The system of claim 41, outputting, via the speech synthesizer or digital articulators, an acoustic speech signal corresponding to the set of generated vocal tract parameters.

43. The system of claim 42, further comprising at least one auditory output device, the speech synthesizer or digital articulators communicating the acoustic speech signal to the at least one auditory output device for audio perception by a subject or other organisms.

44. The system of claim 43, wherein the at least one auditory output device includes at least one earphone, at least one headphone, a tactile transducer, a hearing aid output transducer, a cochlear implant electrode, an auditory brainstem implant electrode, a neural implant electrode, a transcranial electrical stimulation device, or a transcranial magnetic stimulation device.

45. A method of generating sound from muscular appendage movement thoughts, comprising: generating an appendage parameters (AP) to musical instrument parameters (MIP) mapping table; storing, in one or more memory devices of a first computing device the AP to MIP mapping table; receiving and capturing, via at least one electrode array installed or resident in a region of subject’s brain, analog signals, including electrical pulses of varying shapes and amplitudes, representative of muscular appendage movement thoughts, the region of subject’s brain being capable of generating or receiving and understanding the analog signals representative of muscular appendage movement thoughts; communicating, by the at least one electrode array, captured analog signals to the first computing device, the first computing device further including one or more processors, the one or more processors executing computer-readable instructions to: generate appendage parameters from the captured analog signals, wherein the appendage parameters are representative of muscular appendage movement thoughts;translate or convert, utilizing the AP to MIP mapping table, the generated appendage parameters to a set of generated musical instrument parameters, wherein the set of generated musical instrument parameters is representative of sound.

46. An augmentative and alternative communication system, comprising: a first computing device; at least one processor; at least one memory device; computer-readable instructions, the computer-readable instructions executable by the at least one processor to: receive an appendage parameters (AP) to musical instrument parameters (MIP) mapping table; store, in the at least one memory device of the first computing device the AP to MIP mapping table; receive captured analog signals representative of muscular appendage movement thoughts, at the first computing device, the captured analog signals being received from at least one electrode array installed or resident in a region of a subject’s brain, the region of the subject’s brain being capable of generating or receiving and understanding analog signals representative of the muscular appendage movement thoughts; generate appendage parameters from the captured analog signals, wherein the appendage parameters are representative of muscular appendage movement thoughts; and translate or convert, utilizing the AP to MIP mapping table stored in the at least one memory device, the generated appendage parameters to a set of generated musical instrument parameters, wherein the set of generated musical instrument parameters is representative of sound.

47. A method of generating sound from muscular appendage movement thoughts, comprising: generating an appendage parameters (AP) to musical instrument parameters (MIP) mapping table, wherein the appendage parameters are derived from user interactions with an input device; storing, in one or more memory devices of a first computing device the AP to MIP mapping table; receiving and capturing, via the input device, user-generated input signals representative of muscular appendage movement thoughts, which are represented by analog or digital signals, including electrical pulses of varying shapes and amplitudes, wherein the input device includes at least one electrode array configured to capture neuralsignals or a manual input device configured to capture physical appendage movements of the user; communicating, by the input device, captured user-generated input signals to the first computing device, the first computing device further including one or more processors, the one or more processors executing computer-readable instructions to: generate appendage parameters from the user-generated input signals, wherein the appendage parameters are representative of muscular appendage movement thoughts; and translate or convert, utilizing the AP to MIP mapping table, the generated appendage parameters to a set of generated musical instrument parameters, wherein the set of generated musical instrument parameters is representative of sound.

48. An augmentative and alternative communication system, comprising: a first computing device; at least one processor; at least one memory device; computer-readable instructions, the computer-readable instructions executable by the at least one processor to: receive an appendage parameters (AP) to musical instrument parameters (MIP) mapping table; store, in the at least one memory device of the first computing device the AP to MIP mapping table; receive and capture, via the input device, user-generated input signals, which are represented by analog or digital signals, including electrical pulses of varying shapes and amplitudes, wherein the input device includes at least one electrode array configured to capture neural signals representative of muscular appendage movement thoughts or a manual input device configured to capture physical appendage movements of the user; generate appendage parameters from the captured analog signals, wherein the appendage parameters are representative of the muscular appendage movement thoughts or the physical appendage movements; and translate or convert, utilizing the AP to MIP mapping table stored in the at least one memory device, the generated appendage parameters to a set of generated musical instrument parameters, wherein the set of generated musical instrument parameters is representative of sound.

Images