Methods and systems for translation of neural activity into embodied digital-avatar animation

The integration of neural interfaces and machine learning techniques decodes brain activity to restore speech and gesture production in individuals with anarthria and paralysis, overcoming limitations of existing methods by enabling naturalistic and expressive communication.

WO2024254360A1PCT designated stage expired Publication Date: 2024-12-12RGT UNIV OF CALIFORNIA
View PDF 3 Cites 20 Cited by

Patent Information

Application Number
PCT/US2024/032886
Authority / Receiving Office
WO · WO
Patent Type
Applications
Current Assignee / Owner
Priority Date
2023-06-06
Filing Date
2024-06-06
Publication Date
2024-12-12

AI Technical Summary

Technical Problem

Individuals with anarthria and paralysis face challenges in communicating effectively due to limitations in decoding brain activity into naturalistic speech and gestures, with existing methods only enabling text output with limited speed and vocabulary.

Method used

The development of methods and systems that utilize neural interfaces and machine learning techniques to decode brain activity from the sensorimotor cortex, enabling the production of speech sounds, facial movements, and non-speech communicative gestures, using self-supervised learning to discretize actions into meaningful outputs and synthesize them into audio and visual stimuli.

Benefits of technology

Enables full, embodied communication for individuals with severe paralysis by restoring the ability to produce speech and perform gestures, enhancing communication beyond text-based outputs with naturalistic and expressive capabilities.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure US2024032886_12122024_PF_FP_ABST
    Figure US2024032886_12122024_PF_FP_ABST
Patent Text Reader

Abstract

Methods of assisting individuals with communication are provided. In the disclosed methods, cortical activity from a region of the brain associated with movement, speech production, and / or language perception is recorded while an individual attempts to perform an action (e.g., to say words, express an emotion, perform a movement, etc.). Deep learning computational models are used to detect and decode the attempted action from the recorded brain activity. Decoding of actions from brain activity may be aided by the use of self-supervised machine learning techniques, which discretize each action into one or more action representations that serve as an effective intermediary for decoding neural activity patterns and features into meaningful action outputs. Methods for synthesizing decoded actions into audio and / or visual stimuli are also provided, allowing for more naturalistic and expressive communication for individuals who are unable to speak or experience other mobility limitations that inhibit full embodied communication.
Need to check novelty before this filing date? Find Prior Art

Citation Information

Patent Citations

  • Systems and Methods for Decoding Intended Speech from Neuronal Activity

    US20190333505A1

  • Decoding words using neural signals

    WO2011127483A1

  • Methods and devices for real-time word and speech decoding from neural activity

    WO2022251472A1

Cited By

  • Method for predicting CO concentration and coal temperature in spontaneous combustion latency period of coal

    CN120724045A

  • Sign language translation model, system and method based on full-modal alignment

    CN120894834A

  • Performance detection system and detection method for bimodal integrated engine

    CN121026582A

  • Electroencephalogram signal processing method and device and computer equipment

    CN121465609A

  • Depression detection method based on time-frequency-space multi-dimensional feature mining and cross-modal attention fusion

    CN121971093A