Method and apparatus for secure, real-time human-machine interfacing using dense optical muscle imaging

Abstract

Dense optical muscle imaging (dOMG) methods and devices can include wearable interfaces for continuous real-time authentication and control of techniques based on tissue structure and / or motion. The methods and devices described herein can relay individual-specific identification and / or control signals to downstream devices that can be configured to be controlled by the relayed signals, or can be configured to process dOMG data signals to control one or more devices, and / or record, transmit, or analyze information.

Description

[0001] Priority Statement This patent application claims priority to U.S. Provisional Patent Application No. 63 / 601,139, filed November 20, 2023, entitled “METHODS AND APPARATUSES FOR SECURE, REAL-TIME HUMAN COMPUTER INTERFACING USING DENSE OPTICOMYOGRAPHY,” which is incorporated herein by reference in its entirety.

[0002] background Interfacing with digital systems requires a method to verify user authenticity and a means to provide control. Traditional interfaces such as mice, keyboards, or touchpads are agnostic to user identity. To authenticate users, these methods rely on verification methods such as usernames and passwords, or biometric measurements such as fingerprints or facial recognition. However, these methods have several drawbacks that are often exploited: they are static, easily intercepted, and lack the ability to provide real-time, continuous authentication during user interaction with digital systems.

[0003] Public Overview This paper describes methods and apparatus (e.g., systems and devices, including computer software, hardware, and / or firmware) configured to enable wearable human-machine interfaces to provide continuous, real-time authentication and control signals for interactions with various devices. These interfaces utilize densely packed, non-invasive biometric sensors, leveraging the user's unique anatomy and physiological signals controlling arm and finger movements. Unlike traditional biometric technologies such as facial recognition or fingerprints, this approach uniquely identifies the user and decodes their fine motor skills, revealing "who" the user is and "how" they interact with digital devices in real time. This combination provides continuous authentication for every action the user takes using the technology. This method can be used in conjunction with traditional interfaces to authenticate every keystroke, mouse movement, etc., effectively eliminating traditional vulnerabilities associated with usernames and passwords. Furthermore, these methods allow for direct, secure technical control in situations where traditional interfaces are inadequate, such as for people with impaired mobility, or in virtual environments.

[0004] The features and methods described herein may include and improve upon any apparatus and method described in International Patent Application No. PCT / US2023 / 067266, filed May 19, 2023, entitled “TISSUESPECTROPHOTOMETRY FOR HUMAN-COMPUTER AND HUMAN-MACHINE INTERFACING”, which is incorporated herein by reference in its entirety.

[0005] This document describes methods and apparatus for continuous, real-time biometric security and control, which can utilize one or more optical kinesiographic sensors to provide control signals and accompanying identity signals to verify the identity and movement of a user interacting with a device, machine, and / or apparatus. Typically, the methods described herein can non-invasively (i.e., without direct contact, surgical implantation, etc.) detect changes in optical property signals from tissue, such as light absorption, reflection, transmission, optical density, etc., one or more of these. Signals arising from or caused by changes in the optical properties of tissue are referred to herein as “optical property signals.” These optical property signals can be processed to separate specific signals that may indicate specific muscle movements, nerve innervation, and / or movement signals. Additionally, optical property signals can be processed to identify tissue structures and properties, such as the size, orientation, color, and / or texture of skin, scars, tattoos, hair, blood, blood flow, and oxygenation, as well as vascular systems and musculoskeletal structures / systems, including but not limited to muscles, bones, joints, cartilage, tendons, ligaments, blood vessels, and connective elements. These tissue elements can be used collectively and / or individually as unique identifiers of an individual. Therefore, by utilizing appropriate detection density, the same optical property signals can be used to identify individuals through their unique and specific tissue anatomical features, while the time-varying characteristics of tissue optical properties can encode the user's actions and / or anticipated actions. The user's intention can be the user's anticipated action and can include planned movement, imagined movement, or minor physiological changes below a significant movement threshold.

[0006] The methods and apparatus described herein can provide biomedical imaging methods that utilize both static and dynamic optical characteristic signals. Light at visible and near-visible wavelengths (e.g., 650 nm, 900 nm, or one or more wavelengths between about 400 nm and about 1700 nm) is directed at the skin, and varying tissue optical signals are recorded by multiple detectors. Tissues exhibit non-uniform spectral properties depending on their composition. For example, subcutaneous vascular systems (e.g., blood vessels) absorb light differently depending on wavelength (e.g., visible or near-IR light), where veins and arteries can have higher spectrophotometric contrast compared to surrounding tissues. Shorter wavelengths of light are more highly scattered by biological tissues, resulting in tissue features such as skin becoming more prominent. Using high-density image detectors (e.g., imaging arrays or CMOS optical sensors) and / or focusing components (such as conventional lenses, diffusers, or microlens arrays), 1D, 2D, and 3D properties of tissue structures, including skin, vascular systems, tendons, and muscles, can be measured and reconstructed. Much like fingerprints, structural differences exist between individuals, particularly in the vascular system and micro-features of skin texture that can be used for identification. However, voluntary muscle movement causes local tissue deformation (relative to rigid bone), and muscle activity can alter both blood chemistry and flow, collectively leading to changes in tissue optical properties, which are detected by optical methods such as spectrophotometry. Additional changes in tissue optical properties occur due to involuntary variations in blood flow, including cardiac physiology. Both voluntary and involuntary signals generated in some way by muscle movement can be used to infer both specific movement and physiological states of an individual. These signals can be used to authenticate a user and decode their intentions / actions for continuous, real-time control of technologies such as telephones, personal computers, drones, software applications, security systems, etc. In general, a technology that can be authenticated or controlled through this method is hereby referred to as a "downstream device." This paper describes methods and apparatuses that can non-invasively, rapidly, and accurately determine an individual's identity and intentions using optical methods through measurements of tissue movement and / or coarse tissue structure, measurements performed herein in a method referred to as dense optical kinesiography, or dOMG.

[0007] Using one or more (e.g., multiple) optical sensors, both structure- and motion-based variations in organizational optical properties (e.g., reflectivity / absorption / transmittance relative to a fixed location) can be used to create what we refer to in this paper as dense optical motion imaging (dOMG) data. This data can be taken from one or more sensors and can be 1D, 2D, 3D, multispectral, etc., and can be static or time-series. The acquired images are used to create or compare models that can be used to identify individuals, decode identity, intentional / unintentional movement, and the intent to move or act, which can then be used for interaction with technology.

[0008] All the methods and apparatuses described herein (in any combination) are envisioned in this document and can be used to achieve the benefits described herein. Brief description of the attached diagram A better understanding of the features and advantages of the methods and apparatus described herein will be obtained by referring to the following detailed description of illustrative embodiments and the accompanying drawings, in which: Figure 1 A schematic diagram of the wearable device described herein for collecting and synthesizing biometric data from a dense array of optical sensors is shown.

[0010] Figure 2 The illustrations schematically show how the wearable interface described herein can interface with downstream devices (e.g., for control purposes) and remote computing resources (e.g., for data processing).

[0011] Figure 3 The illustration schematically shows sample data collected by placing the device on the wrist, and shows dOMG data from illumination light of different wavelengths, highlighting the anatomical features present.

[0012] Figure 4 An example process for processing data is shown, including, for example, using machine learning (ML) techniques to process / modify raw data and perform data interpretation to construct signals relevant to downstream applications (e.g., identification and / or control).

[0013] Figure 5 The performance of an illustrative machine learning algorithm is shown for determining the wearer's identity and comparing the wearer's real identity with the identity determined (i.e., estimated) by the model.

[0014] Figure 6 An experimental setup for motion decoding of a user with one hand injured and the other uninjured, using a dOMG interface, is shown, including an external tracking camera.

[0015] Figure 7 A schematic diagram is provided illustrating a method for motion decoding of a user with one hand injured and the other hand uninjured using the dOMG interface.

[0016] Figure 8 The process for using the dOMG interface to register and continuously identify users in order to grant them secure access to the password management system is illustrated.

[0017] Figure 9 The process is illustrated for applying dOMG device data to interface with downstream devices while providing continuous authentication.

[0018] Detailed description The dense optical motion imaging (dOMG) methods and apparatus described herein may include wearable interfaces for continuous, real-time authentication and control of tissue structure- and / or motion-based technologies. These methods and apparatuses may relay individual-specific identification and / or control signals to downstream devices that can be configured to be controlled by the relayed signals, or to process dOMG data signals to control one or more devices, and / or record, transmit, or analyze information.

[0019] For example, the apparatus described herein may include: a plurality of optical sensors configured to sense optical properties, wherein the plurality of optical sensors include at least one light emitter and a plurality of optical detectors; a support configured to hold the plurality of optical detectors of a spectrophotometer near a skin surface such that the plurality of optical detectors are arranged in a pattern relative to the skin surface; and a processor configured to receive signals from the plurality of optical detectors. These various dOMG data streams are used to separate anatomical features associated with skin, neuromuscular, vascular, and other tissues (e.g., moles, pores, hair, etc.), and to track physiological changes in anatomical structures that may occur over time in relation to phenomena such as movement, aging, etc. The separation of anatomical features and / or changes in these features over time may be performed optically (e.g., by adjusting illumination, optical sensors, and / or other hardware / imaging parameters) and / or using statistical / machine learning techniques (e.g., signal processing, dimensionality reduction, neural networks, etc.). Static and / or time-series data, including various types of raw signal data, reconstructed image data, and / or (e.g., data transformed by passing the data through statistical and / or machine learning models), can be used to train one or more machine learning algorithms to achieve one or more downstream applications (e.g., user identification, decoding of user motion, control of mechanical equipment, etc.).

[0020] Figure 1 An example of the apparatus as described herein is shown. Figure 1In this device 100, a support 101 is configured as a strap or band that can be secured to a subject's arm, forearm, wrist, etc. The support 101 supports multiple dOMG sensor arrays along the inner side (skin-facing side) of the band. Each dOMG sensor array includes an optical emitter and a detector 105. The sensor arrays may be integrated into the support or coupled to it. The device 100 also includes a processor 107 that includes or is coupled to an output terminal 109. The processor and / or output terminal may be located within a housing attached to the support. In some examples, optical characteristic signals may be measured by the system (e.g., by the optical sensor arrays). In some examples, optical elements 103, such as lenses or multiple lenses, diffusers, or microlens arrays, may be placed between the skin and the sensors. The optical elements may be custom-designed and constructed using additive manufacturing or similar methods.

[0021] Figure 2 An example of a dOMG interface or subsystem as described herein is schematically illustrated, wherein an optical system can be used to interface with downstream devices via continuous real-time authentication and control. The dOMG interface system 201 includes one or more dOMG sensor groups having both a transmitter and a sensor that provides input to a processor (e.g., a pair or combination of transmitter / sensor including an optical transmitter and an optical sensor). In this example, the wearable interface 201 can be configured to be worn as a strip, band, garment, brace, patch, etc., adjacent to or against the subject's skin. The system also includes one or more optical sensor groups 203 (in addition to other sensors (e.g., IMU, etc.) 205) as described herein and processing circuitry 207 for connecting / controlling / coordinating the sensors and processing the raw signals from the sensors. The processing circuitry 207 and / or processor 209 may include modules for processing the signals (e.g., filtering, amplifying) and decoding the intended action and / or user identity. Optionally, processing circuits 207, 209 can communicate with another (auxiliary) processor 211, which can store, transmit (e.g., to a remote or local server 213), or process data from the sensors. The processing circuitry may include a flexible printed circuit board (PCB) or microwire interconnects, and a microprocessor that provides power and common ground to the sensors. The microprocessor may also provide signal processing. Optionally, processing can be performed directly by processor 209 (e.g., a microprocessor, computer, etc.) without intermediate circuitry. Data recorded from the sensors can be streamed to the processor and other devices 211 for authentication and / or closed-loop control.

[0022] These devices (e.g., systems, devices, etc., including software, hardware, and firmware) can be wearable systems for measuring tissue structure and movement. Typically, an optical sensor array can be configured to detect optical properties from tissue and may include, for example, one or more light emitters and one or more optical detectors. The light emitter can be any suitable light emitter, such as, but not limited to, LEDs, lasers, etc. The light emitter can emit a single wavelength or color, a certain wavelength range, or multiple different discrete wavelengths (or discrete wavelength bands). For example, the light emitter may include an LED configured to emit red light. In some examples, the light emitter can be configured to emit light in the infrared (IR, including near-infrared) spectrum, such as light between about 700 nm and about 800 nm. In some examples, the light emitter can be configured to emit light between about 600 nm and about 990 nm. In some examples, the light can be emitted continuously or in a pulsed manner (e.g., at frequencies between about 5 Hz and 1000 Hz, greater than 10 Hz, greater than 100 Hz, etc.). The light emitter may include multiple light emitters configured to emit at two or more different wavelengths or wavelength ranges. The light emitter may include a light source and a medium through which it passes, such as air, polymer, plastic, glass, gel, and / or some combination thereof. The light emitter may emit directional light at an angle of incidence ranging from 0 degrees (parallel) to 90 degrees (perpendicular) relative to the imaging plane, or some combination thereof. The optical detector may be any suitable optical detector, such as (but not limited to) a photodetector / photosensor, such as a photodiode, charge-coupled device (CCD), phototransistor, quantum dot photoconductor, photovoltaic device, photochemical receptor, neuromorphic imager, etc.

[0023] Typically, optical sensor arrays can be integrated, such that one or more light emitters are paired with one or more light sensors. In some examples, an optical sensor array may include a single light emitter or a pair of light emitters and multiple light sensors. In some examples, one or more light emitters may be separate from one or more light sensors.

[0024] One or more light emitters and one or more light sensors may be arranged and / or fixed to a support such that the light emitters and light sensors of the optical sensor group are arranged adjacent to each other and / or opposite to each other, such that light from the light emitters can travel through the tissue before the light sensors sense the light.

[0025] The methods and apparatus described herein can typically detect optical properties of tissue that may be related to muscle movement. For example, as will be described herein, optical properties may be the absorption, emission, and / or reflection of light by the tissue. Figure 3Examples of tissues that can be examined via this method are shown. A schematic diagram of a user's left forearm and hand 301 is shown here. The illustrations show some non-exhaustive examples of biomarkers that contribute to the optical properties of tissues, as observed with light of different wavelengths (green 303 and red 305), including skin texture 307, tendons 309, skin ridges 311, blood vessels 313, hair 315, etc. The time-varying variations of these anatomical features can be used as control signals specific to a particular individual.

[0026] The methods or apparatus described herein may include one or more processors, such as microprocessors and / or additional circuitry. The processor may include instructions for performing any of the methods described herein. Specifically, the processor may be configured to separate components of the optical characteristic signal corresponding to heartbeat, voluntary or involuntary muscle movement, or any or all tissue characteristics, such as the size, orientation, color, and / or texture of skin, hair, blood, vascular system, and musculoskeletal structures / systems, including but not limited to muscles, bones, joints, cartilage, tendons, ligaments, and connective tissue. The optical characteristics may correspond to differential optical characteristic signals of light at two (or more) wavelengths. The processor may be configured to compare any detected optical characteristic signal with previously acquired optical characteristic signals to determine the probability that these signals originate from the same individual. Therefore, the processor may use statistical and / or machine learning (ML) methods, such as artificial neural networks (e.g., convolutional neural networks). Additionally, the optical sensor array and the processor may consistently perform methods to remove, separate, or enhance high spatial frequency information and / or low spatial frequency information, ambient light, noise, and / or artifacts from an image.

[0027] Typically, optical sensor assemblies are attached to a support. For example, the device may include multiple optical sensor assemblies secured by a support. The support can be any structure configured to hold the optical sensor assemblies near tissue from which signals will be measured non-invasively. The support can be coupled to some or all of the light emitters, optical components, optical detectors, sensors, microprocessors, and circuitry used in a fully functional device. For example, the support can be a flexible wristband made of one or more flexible, close-fitting, and / or stretchable materials (i.e., fabric, elastic materials, plastics, silicone, flexible metals). A portion of the support can be made of a rigid material (i.e., plastic, metal, glass, ceramic) to accommodate non-flexible device components. For example, the support can be an accessory to existing wrist wearables (such as watches, smartwatches, fitness trackers, or jewelry items) and take the form of existing components of said wearables (i.e., watch straps), or an auxiliary component. The support may include additional flexible materials (such as foam, plastic, silicone) at its interface with the skin to help block ambient light. Any of these structures or supports can be manufactured using additive manufacturing (e.g., 3D printing), injection molding, or similar manufacturing methods. In one example, the support may be or may include other wearable structures on clothing, watches, or a user's arm (e.g., forearm or wrist). For example, the support may be or may include strips, bands, or patches. In some examples, the support may be configured to be attached to the body (and in some cases removably attached to the body). In some examples, the support is configured to fit onto a user's arm (e.g., forearm, shoulder, upper arm, wrist, elbow, hand, and / or rings, etc.). An optical sensor array may be coupled to the support. For example, the optical sensor array may be rigidly coupled to the support and / or flexibly coupled to the support. In addition to optical sensors, the support may also hold processors (e.g., controllers, control circuitry, microprocessors, electronic communication circuitry, memory, etc.) and / or power sources (e.g., batteries, capacitive power supplies, regenerative power supplies, etc.) and / or connectors (e.g., wires, traces, etc.). The support may hold other non-optical sensors (e.g., IMU, GPS, temperature, force / pressure, fingerprint) and / or interface components (e.g., mechanical or capacitive buttons, dials, and sliders) and / or communication components (e.g., microphones, speakers, haptic motors). The support may include one or more housings that enclose all or part of the aforementioned components.

[0028] Any of the devices (apparatus, systems, etc.) described herein may also include one or more signal conditioners configured to modify (e.g., regulate) signals from or derived from an optical sensor array. For example, a signal conditioner may include one or more lenses, diffusers, filters, and lens arrays. The signal conditioner may be part of or separate from the optical sensor array. The signal conditioner may be coupled to a support and / or at least partially enclosed within a housing. In some examples, regulation may also be partially performed by a processor.

[0029] In any of these devices or methods, the processor can be configured to distinguish tissue movement patterns or static characteristics of tissue as individual-specific. The processor can be configured to separate optical signals corresponding to heart rate, respiratory rate, or any other regular periodic signals from received optical characteristic signals by acquiring signals common to multiple optical detectors.

[0030] A processor includes hardware that runs computer program code. Specifically, the term "processor" can include (or be part of) a controller, and can encompass not only computers with different architectures (such as single-processor or multi-processor architectures and sequential (Von Neumann) or parallel architectures) but also specialized circuitry such as field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), signal processing devices, and other devices. A processor can include one or more of the following computing elements: one or more microprocessors integrated as part of a device (i.e., coupled to a support structure); a computer, telephone, or computing device connected to the device (e.g., via Bluetooth, Wi-Fi, USB, RF, or other similar means); or computing resources (i.e., a remote server or virtual machine). These processors can operate independently and / or in combination with each other to perform operations related to the synthesis, storage, interpretation, and application of data. For example, the transformation of raw data and the application of ML models produce continuous output signals reporting the wearer's identity and / or intent. In another example, a supervised ML model generates concurrent authentication and control signals that are passed to downstream devices (e.g., personal computers, telephones, remotely controlled electronic or mechanical devices, etc.).

[0031] Figure 4An example of a process for processing dOMG and other sensor data is shown. In this example, the raw (unprocessed) dOMG and other sensor outputs 401 can be modified during the preprocessing stage, for example, for amplification, filtering, signal subtraction (e.g., heartbeat signal subtraction), digitization, etc. 403. Preprocessing can be integrated with the dOMG sensor array itself, or it can be separate. In some examples, preprocessing can be integrated with a processor, or preprocessing can be coupled to a processor but separate from the processor. In some examples, preprocessing may include registering optical sensors (and / or registering dOMG representations obtained by optical sensors) 405. Processing can be performed locally by an onboard processing unit and / or can be transmitted via a transmitter for remote processing. In either case, optical data encoding the user's anatomical and physiological structures includes identifying how signals of position and / or motion and / or optical characteristics of body regions should be interpreted as control or identification signals for interfacing with devices, computers, and / or software that receive output indicators of the position and / or motion of body parts (or indicators of that motion) and user identity.

[0032] As described above, the computation supporting the synthesis and interpretation of signals can occur partially and / or entirely on the wearable device (i.e., on an embedded microprocessor), or partially or entirely remotely (e.g., using downstream devices or remote processors located in separate remote computing environments). The synthesis of the derivative stream used to infer the wearer's identity and / or control state can occur on the embedded microprocessor, and the derivative stream is generated as output from a statistical and / or ML model trained using remote data and computing resources (i.e., on downstream devices, remote servers, cloud environments, etc.). In this example, the device streams data to a local computer, which then transmits the data to a remote computing environment, triggering the creation of a machine learning (ML) model. Alternatively or in combination, the model can be trained locally on a processor with a dOMG interface without access to downstream devices or servers. Once completed, the ML model is packaged as software containing a complete set of instructions for converting the raw dOMG and sensor data collected or generated by the device into an output stream suitable for implementing identity and / or control applications. In this example, the ML model generated in a remote computing environment is then transmitted (e.g., via the Internet) to a local computer, and then (e.g., via Bluetooth, wireless, etc.) to the microprocessor itself. The output stream from the data synthesis is then passed (directly or via an application programming interface or graphical user interface) to software and hardware tools to enable applications including authentication, verification, control, etc. Examples of output streams generated by devices (e.g., via raw data collection and / or via transforming data using the ML model) include, but are not limited to, control signals, keystrokes, mouse movements, wearer identity or body position, certain dOMG signals or signatures, raw and / or processed dOMG or sensor data. Data streaming and synthesis can be performed continuously and / or in real time, or can be stored for later transmission, review, analysis, and use, including improving the performance of the ML model used for the applications described herein. Processed or raw data can be stored on the dOMG interface and / or on downstream devices, remote servers, etc.

[0033] In any of these devices or methods, the system can incorporate a self-evaluation mechanism to determine the presence of a wearer and, if worn, the details of the interaction between the wearer and the downstream device. In one example, it can determine whether the device is currently being worn and whether the fit is appropriate (i.e., neither too loose nor too tight). Furthermore, the system can assess proximity to the downstream device and whether the device is within a predefined range for communication. If the system detects that it is too far from the downstream device, it can take predetermined actions, such as warning the user or initiating security protocols to prevent access in the event of physical separation and / or unauthorized use. These assessments can leverage an integrated sensor array to decode its operational state in real time, including but not limited to: dOMG sensors, inertial measurement units (IMUs) and wireless modules (e.g., Bluetooth, Wi-Fi, RF, etc.), temperature sensors, pressure sensors, force sensors, position / tracking sensors (GPS, etc.), and / or capacitive sensors.

[0034] One or more cameras can be used to track body movement, allowing them to coordinate with detected optical signals and signals from other sensors (e.g., IMU) on the worn device. For example, the device can be used to train body reconstruction using computer vision during and / or after a calibration phase, utilizing one or more cameras associated with the signals. In some examples, these cameras can be worn on the body, such as virtual reality, augmented reality, or mixed reality (collectively referred to as extended reality, XR) headsets or smart glasses. In some examples, instead of a camera observation system or in addition to one, the device can use depth or force-sensitive sensors to train body reconstruction, such as IR dot arrays or surface areas for contact-based force measurements, using a touchscreen or camera and a transparent surface. In some examples, the device can train a motion / interface decoder based on anticipated or indicated actions. In some cases, the device can train a motion / interface decoder based on user feedback / input via self-reporting and / or using indicated reports. In other cases, the camera can use pose estimates of body parts and correlate those pose estimates with optical characteristic signals. Traditional computer interfaces can also be used to infer body position, including key presses, computer mouse movements and clicks, touchscreen input, etc.

[0035] In any of the methods and apparatuses described herein, processing and synthesizing data to generate necessary insights can utilize statistical and / or machine learning methods (e.g., convolutional neural networks, regression, clustering, decomposition). These statistical and / or machine learning models may include supervised methods where real-world data about user identity and / or actions is available during training, and / or may include unsupervised methods where such real-world information is not available to the model. Data collection for implementing supervised learning models may involve prompting the user to perform various body movements. In some examples, gestures and postures may be performed. In some examples, the user may interact with external objects, including but not limited to force sensors, keyboards, mice, touchscreens, remote control devices, and non-electronic objects. In some examples, the user may interact with virtual objects in XR. In some examples, the user may use a separate data collection device (such as a smartphone camera) to provide biometric information, with or without a dOMG interface. In all these examples, the user may perform movements spontaneously or guided by visual, tactile, and / or auditory commands. In all these examples, the user may perform movements consciously, involuntarily, or mechanically assisted (e.g., using a translation stage or robotic exoskeleton). Data collected from one or more users via any of the examples above can involve creating ML and / or statistical models. Models can be trained on data generated from one or more sessions, such as data from one or more users and one or more wearable devices, to produce models that are generalizable across users and / or devices. Outputs relevant to one or more applications can be derived from a single model or from multiple ML models (e.g., hierarchical models, ensemble models, transfer learning, etc.) using combinations of different data sources, model formulas, or objective functions. For example, data collected from multiple users can be used to train a model for extracting features that distinguish individuals present in the training data; the same model can then be tuned (e.g., using statistical methods and / or transfer learning) to produce an output stream that demonstrates whether / when a device is worn and whether / when the wearer's identity matches the identity of the individual to whom the device is registered. Figure 5 The accuracy performance of the ML model in identifying which user is wearing the device is shown here as a confusion matrix. The y-axis represents the user's true identity, while the x-axis represents the identity determined by the ML model. The model demonstrates high accuracy, specificity, and precision across all users tested in this example.

[0036] Data collected from devices (dOMG and / or sensor data) and / or externally collected data can undergo dimensionality reduction methods such as principal component analysis, singular value decomposition, nonnegative matrix factorization, clustering, etc., with the aim of extracting relevant physical and conceptual features, both static and dynamic. In one example, real-body tracking data captured by a webcam can be in the form of node locations in 3D space, which can be reduced to a set of standardized movements or gestures using dimensionality reduction methods.

[0037] Reapplying a previously trained decoder Leveraging a user's unique anatomy, previously created ML or statistical models can be applied to a variety of wearable scenarios. This is accomplished by determining the position and orientation of the system / device on the user's body relative to when the device was previously worn. Using computer vision and image registration algorithms, positional information can be inferred from biomarkers (such as vascular systems, hair, pores, skin ridges and wrinkles, moles and blemishes) and / or artificial marks (such as tattoos, stickers, or marks drawn / placed on the skin). Alternatively or in combination, position sensor signals collected by an IMU can be used to measure the system's position relative to the user's body.

[0038] Efforts can be made to make the model robust to changes in the device's position or placement during wear. This can be done to account for loosely worn devices or to compensate for changes in the system's physical location relative to body tissue. In one example, an array of optical sensors or an external imaging device with a larger physical extent than the system can be used to collect data from a body area much larger than the area of ​​the wearable device. In another example, a model can be created using data from a variety of combinations of wear locations.

[0039] Adaptive model After a model is created for a user, additional dOMG data may be collected over subsequent days, weeks, and / or years for the purpose of updating the statistical and / or machine learning model. This additional dOMG data can be used to create new models or update existing models to improve the accuracy / fidelity of user identification and / or body state inference. This data may include feedback from the user and / or, on an ongoing basis, incorporate behavioral patterns and interactions with downstream devices. In some cases, the model may utilize spontaneous and / or indicated body positions, movements, and gestures to collect dOMG data to update or recreate the ML model. The purpose of this process is to incorporate gradual or abrupt anatomical changes that may occur in the wearer, such as skin tightness, tone, or blemishes, for example due to age, makeup, physical activity, illness, and / or intentional or unintentional tissue modifications that may occur over time (e.g., tattoos, makeup, incisions, scars). In some examples, the model may be updated to reflect changes, improvements, or modifications in the dOMG device used to create the ML model (e.g., changes in calibration or physical hardware). The process for data collection and model updates can use computers and / or storage resources located on the dOMG device (i.e., an embedded processor), a local computer (e.g., wirelessly connected to the device), or a remote server (e.g., in the cloud), where any previously collected data can be referenced.

[0040] Application in prosthesis control of motion-impaired users The methods and apparatus described herein can be used for any suitable application, including medical applications. For example, one application is to provide control / identity signals to users with mobility impairments. Such impairments may be due to any injury or disability resulting in any form of reduced mobility, including paraplegia, amputation, deformity, nerve damage or injury, such as ALS, Parkinson's disease, etc. In one example, the user may have an impairment on one side of their body, where movement and body position on the unimpaired side can be used to estimate the expected movement on the impaired side. Figure 6 An experimental setup for motion decoding using a dOMG device is shown. One hand is impaired (i.e., its range of motion and / or dexterity is limited) and the other hand is unimpaired. The dOMG device 601 is worn on both wrists of the user, while a tracking camera 603 focuses on the movements made by the unimpaired hand 605 and the impaired hand 607. Figure 7A method for motion decoding of a user with one hand injured and the other uninjured is described using a dOMG interface 701. The user can be instructed to (attempt) perform symmetrical, identical movements 703 with both hands while dOMG signals and hand-tracking information 705 are collected. Therefore, the relationship between the dOMG signal on the injured side 709 (which, due to injury, does not cause the wearer's expected movement) and the actual expected finger movement can be inferred using hand and finger movements from the uninjured side 707. In another example, the user can be instructed to attempt or imagine performing certain movements or actions, and their dOMG data can be used to estimate their expected movement. In any of these examples, the relationship can then be used for any of the aforementioned control purposes, as detailed in the following sections. In other cases, a device equipped with inferred knowledge of the location of the user's injured hand and fingers can provide feedback that can be used to guide physical therapy.

[0041] Continuous real-time password management As detailed above, biometric data collected by the device, combined with statistical and machine learning models used to synthesize the data, provides a mechanism for continuously verifying whether / when the device is worn, identifying the wearer, and decoding the actions they perform. The model performing any and all of these functions is personalized, thus reporting not only the individual's actions but also the identity of the individual performing the actions.

[0042] Figure 8 A process for using a dOMG device to register and continuously identify users to grant secure access to a password management system is illustrated. A new user (e.g., by granting access to the password manager) enters their name and login credentials 801 and is guided through a series of prompts designed to generate biometric data 803 for training a feature extraction and / or machine learning model to uniquely identify a registered user based on their biometric data 805. The registration process produces a user identification model artifact containing software instructions for processing the biometric data to determine whether the wearer's identity matches that of a registered user, a previously registered user, or whether the identity is unknown 807. In identification mode, the dOMG device continuously streams raw and processed biometric signal data to downstream devices (e.g., computers, telephones, etc.) 809. The streaming data communicates to the downstream devices whether the dOMG device is being worn. If it is determined at any time that the device is not being worn, the user logs out 811. If the device is being worn, software is executed using the identification model artifact to verify the wearer's identity 813. If identity cannot be verified, access is revoked and unauthorized use (815) can be reported. If user identity is verified, access to the trusted password management system (819) is granted (817).

[0043] Combining the model output used by password management services (e.g., 1Password, Bitwarden, iCloud Keychain, etc.) can provide alternatives to single-factor or multi-factor authentication, where dOMG data collected from the wearer can be used (e.g., using the statistical and machine learning methods described above) to uniquely identify the wearer. Continuously providing the wearer's identity can act as a certificate, ensuring that an individual seeking access to an account is indeed authorized to have such access (e.g., using password management software to determine which accounts a user can access). In other examples, data input to downstream devices is "watermarked" or attributed to a specific individual (e.g., individual keystrokes in an email may be attributed to a specific person or group). This can be used as a means of verifying the origin of digital content.

[0044] As the amount of data generated during daily use grows, a more data-intensive model for decoding wearers' actions (and simultaneously verifying their identity) can act as a layer of protection against password theft and unauthorized use. Initially, this can work by automatically logging out of the service or reporting unauthorized use when the wearer's identity changes or when the authenticity of a request cannot be verified. Ultimately, real-time authentication reported by the device can protect any and all interactions within a secure context.

[0045] Continuous real-time attestation of human-machine / device / machine interaction The methods and apparatus described herein provide a mechanism for verifying the identity of a wearer while using supervised and / or unsupervised learning to decode motion states (e.g., hand / wrist / finger movements / positions) to control downstream devices (e.g., computers, electronic devices, mechanical devices, etc.). These concurrent signals (user identity and neuromuscular decoding) can be used to continuously verify the identity of the wearer issuing certain control actions, to verify that the user is authorized to perform certain actions, and to maintain a continuous record of the identity of the individual responsible for requesting or issuing certain actions.

[0046] Figure 9A process is illustrated that uses dOMG device data to interface with downstream devices while providing continuous authentication. During registration, the user is prompted to perform a series of specific actions (e.g., movement and / or control) while wearing the device 901; dOMG data is recorded while these actions are performed 903. The data generated from 901 and 903 is then used to train one or more models 905 for user identification and decoding. The dOMG data generated during use is then used to continuously verify the wearer's identity, and if the user's identity is unknown or incorrect, intervention measures are taken (e.g., reporting unauthorized use) 907. If identity is verified, the decoding model is applied to the dOMG data to infer the user's actions / intent 909, and control signals are sent to the downstream device 911. The decoding and / or identification models 913 can be periodically updated using data generated during device use.

[0047] The systems and associated methods described herein can be applied to control systems operating on devices, machines, computers, and / or computerized devices, and to interact with systems operating on such devices, machines, computers, and / or computerized devices. In some examples, the body state inference techniques described herein can be used to interface with a computer, where a user mimics physical movements typically associated with conventional input devices (i.e., mouse, keyboard, touchpad, touchscreen, etc.) without these physical devices, and these movements are translated into computer input. In some examples, the system can use the aforementioned body state inference techniques to map body states not typically used for computer control to computer input, such as bending or straightening fingers to move a cursor, pressing the thumb and fingers together with gradient force to select from a drop-down menu, flicking fingers to change slides in a slideshow, and so on. In some examples, the system can use the aforementioned body state inference techniques to map body states to computer input, but gradually change the mapping of body states to computer input over time, allowing a user with repetitive stress disorder to repeatedly perform the same computer input while physically moving their body differently each time. In some examples, the system can use the aforementioned body state inference techniques to determine body positions and / or movements that are considered unhealthy or high-risk for injured or healthy users, and suggest alternative input mappings for safer and more intuitive computer control.

[0048] Secure direct human-machine interfacing The dOMG interface can be used to interact with any kind of remotely controlled machine or device. Machines can include, but are not limited to, electronic devices such as televisions, drones, robots, equipment, vehicles, or other mechanical devices. Control signals can be inferred from biometric signals and / or other signals (e.g., IMU measurements) recorded by the device, for example, by using supervised learning to train statistical and / or ML models, as described above. Control signals can be transmitted to downstream electronic or mechanical devices via one or more communication modes, including but not limited to issuing direct control signals, sending control signals to a graphical user interface, and sending control signals to an application programming interface (API), which then interprets and issues downstream control signals. As with the other applications detailed above, control signals can be generated concurrently with signals verifying the wearer's identity, ensuring that docking is performed by an authorized user who can exercise the necessary controls.

[0049] Controlling an extended reality interface The systems and associated methods described above can be applied in contexts of virtual reality, augmented reality, and / or mixed reality technologies. In some examples, the system can use the aforementioned body state inference techniques to provide a virtual representation of a user's body, hands, and / or fingers on one or both sides of the body in an XR context, typically used in interface control scenarios. In some examples, the system can use the body position inference techniques described herein, combined with wearable or external body tracking devices (such as headsets, smart glasses, motion capture equipment, etc.), to improve the performance accuracy of said devices in situations where other devices may fail, such as when tracking is obstructed by another body part or the environment, or when information that other body tracking methods cannot retrieve (such as pressure applied by fingers) is needed in an XR context. In some examples, the system can infer the identity of the user of an XR device (such as a headset or smart glasses), enabling only authorized personnel to participate in specialized XR environments and / or experiences, such as private meeting rooms, or to participate in customized settings based on wearer preferences.

[0050] All disclosures and patent applications mentioned in this specification are incorporated herein by reference in their entirety, to the extent that each individual disclosure or patent application is expressly and individually indicated to be incorporated herein by reference. Furthermore, it should be understood that all combinations of the foregoing concepts and other concepts discussed in more detail below (provided these concepts are not inconsistent with each other) are considered part of the inventive subject matter disclosed herein and can be used to achieve the benefits described herein.

[0051] Any of the methods described herein (including the user interface) can be implemented as software, hardware, or firmware, and can be described as a non-transitory computer-readable storage medium storing a set of instructions executable by a processor (e.g., a computer, tablet, smartphone, etc.), which, when executed by the processor, causes the processor to perform any of the steps, including but not limited to: displaying, communicating with a user, analyzing, modifying parameters (including timing, frequency, intensity, etc.), determining, alerting, or similar steps. For example, any method described herein can be performed at least in part by a device including one or more processors having a memory storing a set of instructions for the procedures of the method.

[0052] While various embodiments have been described and / or illustrated herein in the context of a full-featured computing system, one or more of these exemplary embodiments may be distributed as a program product in various forms, regardless of the specific type of computer-readable medium on which the distribution is actually performed. The embodiments disclosed herein may also be implemented using software modules that perform certain tasks. These software modules may include scripts, batch files, or other executable files that may be stored on computer-readable storage media or in a computing system. In some embodiments, these software modules may configure a computing system to perform one or more of the example embodiments disclosed herein.

[0053] As described herein, the computing devices and systems described and / or illustrated herein broadly refer to any type or form of computing device or system capable of executing computer-readable instructions, such as those contained within the modules described herein. In their most basic configuration, each of these computing devices may include at least one memory device and at least one physical processor.

[0054] As used herein, the term "memory" or "memory device" generally refers to any type or form of volatile or non-volatile storage device or medium capable of storing data and / or computer-readable instructions. In one example, a memory device may store, load, and / or maintain one or more modules described herein. Examples of memory devices include, but are not limited to, random access memory (RAM), read-only memory (ROM), flash memory, hard disk drive (HDD), solid-state drive (SSD), optical disk drive, cache, variations or combinations thereof, or any other suitable storage memory.

[0055] Furthermore, as used herein, the term "processor" or "physical processor" generally refers to a processing unit of any type or form of hardware implementation capable of interpreting and / or executing computer-readable instructions. In one example, a physical processor may access and / or modify one or more modules stored in the aforementioned memory devices. Examples of physical processors include, but are not limited to, microprocessors, microcontrollers, central processing units (CPUs), graphics processing units (GPUs), field-programmable gate arrays (FPGAs) implementing soft-core processors, application-specific integrated circuits (ASICs), portions of one or more of these, variations or combinations thereof, or any other suitable physical processor.

[0056] Although shown as separate elements, the method steps described and / or illustrated herein may represent portions of a single application. Furthermore, in some embodiments, one or more of these steps may represent or correspond to one or more software applications or programs that, when executed by a computing device, enable the computing device to perform one or more tasks, such as the method steps.

[0057] Furthermore, one or more of the devices described herein can convert data, physical devices, and / or representations of physical devices from one form to another. Additionally or alternatively, one or more modules described herein can convert a processor, volatile memory, non-volatile memory, and / or any other part of the physical computing device from one form of computing device to another by executing on a computing device, storing data on a computing device, and / or otherwise interacting with a computing device.

[0058] As used herein, the term "computer-readable medium" generally refers to any form of device, carrier, or medium capable of storing or carrying computer-readable instructions. Examples of computer-readable media include, but are not limited to, transmissive media (such as carrier waves) and non-transitory media such as magnetic storage media (e.g., hard disk drives, magnetic tape drives, and floppy disks), optical storage media (e.g., optical discs (CDs), digital video discs (DVDs), and Blu-ray discs), electronic storage media (e.g., solid-state drives and flash memory media), and other distribution systems.

[0059] Those skilled in the art will recognize that any process or method disclosed herein can be modified in various ways. The process parameters and order of steps described and / or illustrated herein are given by way of example only and can be changed as needed. For example, while the steps shown and / or described herein may be shown or discussed in a particular order, these steps do not necessarily need to be performed in the order shown or discussed.

[0060] The various exemplary methods described and / or illustrated herein may omit one or more steps described or illustrated herein, or include additional steps in addition to those disclosed herein. Furthermore, steps of any method disclosed herein may be combined with any one or more steps of any other method disclosed herein.

[0061] The processor described herein can be configured to perform one or more steps of any of the methods disclosed herein. Alternatively or in combination, the processor can be configured to combine one or more steps of one or more methods disclosed herein.

[0062] When a feature or element is referred to herein as "on another feature or element," it may be directly on the other feature or element, or there may be intermediate features and / or elements present. In contrast, when a feature or element is referred to as "directly on another feature or element," there are no intermediate features or elements. It will also be understood that when a feature or element is referred to as "connected," "attached," or "coupled" to another feature or element, it may be directly connected, attached, or coupled to that other feature or element, or there may be intermediate features or elements present. In contrast, when a feature or element is referred to as "directly connected," "directly attached," or "directly coupled" to another feature or element, there are no intermediate features or elements. Although described or illustrated with respect to one embodiment, the features and elements thus described or illustrated can be applied to other embodiments. Those skilled in the art will also recognize that references to structures or features positioned "adjacent" to another feature may have portions covering or located below the adjacent feature.

[0063] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the invention. For example, as used herein, the singular forms “a,” “an,” and “the” are also intended to include the plural forms unless the context clearly indicates otherwise. It will be further understood that, when used in this specification, the terms “comprises” and / or “comprising” specify the presence of stated features, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, elements, components, and / or groups thereof. As used herein, the term “and / or” includes any one and all combinations of one or more of the associated listed items and may be abbreviated to “ / .”

[0064] Spatially relative terms such as “under,” “below,” “lower,” “over,” “upper,” and similar terms may be used herein to readily describe the relationship of one element or feature to another (or more) elements or features, as illustrated in the accompanying figures. It will be understood that spatially relative terms are intended to encompass different orientations of the device in use or operation, other than those depicted in the figures. For example, if the device in the figures is inverted, an element described as “below” or “under” other elements or features would be oriented as “above” other elements or features. Thus, the exemplary term “under” can encompass both above and below orientations. The device may be oriented in other ways (rotated 90 degrees or in other orientations), and the spatially relative descriptive terms used herein are interpreted accordingly. Similarly, the terms “upwardly,” “downwardly,” “vertical,” “horizontal,” and similar terms are used herein for illustrative purposes only, unless otherwise specifically indicated.

[0065] While the terms "first" and "second" may be used herein to describe various features / elements (including steps), these features / elements should not be limited by these terms unless the context otherwise requires. These terms may be used to distinguish one feature / element from another. Therefore, without departing from the teachings of the invention, the first feature / element discussed below may be referred to as the second feature / element, and similarly, the second feature / element discussed below may be referred to as the first feature / element.

[0066] Generally, any of the apparatuses and methods described herein should be understood as inclusive, but all or a subset of the components and / or steps may optionally be exclusive and may be expressed as “consisting of various components, steps, sub-components or sub-steps” or optionally “consisting substantially of various components, steps, sub-components or sub-steps”.

[0067] As used herein in the specification and claims, including in the examples, and unless otherwise expressly specified, all figures may be considered as if preceded by the words “about” or “approximately,” even if the term is not explicitly stated. The phrase “about” or “approximately” may be used when describing magnitude and / or location to indicate that the described value and / or location is within a reasonably expected range of value and / or location. For example, numerical values ​​may have values ​​as + / -0.1% of the stated value (or range of values), + / -1% of the stated value (or range of values), + / -2% of the stated value (or range of values), + / -5% of the stated value (or range of values), + / -10% of the stated value (or range of values), etc. Unless the context otherwise indicates, any numerical value given herein should also be understood to include about or approximately that value. For example, if the value “10” is disclosed, then “about 10” is also disclosed. Any numerical ranges listed herein are intended to include all subranges contained therein. It should also be understood that when a value is disclosed, "less than or equal to" that value, "greater than or equal to" that value, and possible ranges between values ​​are also disclosed, as would be appropriately understood by a person skilled in the art. For example, if the value "X" is disclosed, then "less than or equal to X" and "greater than or equal to X" (e.g., where X is a numerical value) are also disclosed. It should also be understood that throughout the application, data is provided in a variety of different formats, and that the data represents the endpoints and starting points and ranges of any combination of data points. For example, if a specific data point "10" and a specific data point "15" are disclosed, it should be understood that greater than, greater than or equal to, less than, less than or equal to, and equal to 10 and 15, as well as 10 to 15, are disclosed. It should also be understood that each unit between two specific units is also disclosed. For example, if 10 and 15 are disclosed, then 11, 12, 13, and 14 are also disclosed.

[0068] While various illustrative embodiments have been described above, any of several changes may be made to the various embodiments without departing from the scope of the invention as described in the claims. Optional features of the various device and system embodiments may be included in some embodiments but not in others. Therefore, the foregoing description is provided primarily for illustrative purposes and should not be construed as limiting the scope of the invention as set forth in the claims.

[0069] The examples and illustrations included herein show specific embodiments in which the subject matter can be practiced by way of illustration and not limitation. As mentioned, other embodiments can be utilized and derived therefrom, making structural and logical substitutions and changes possible without departing from the scope of this disclosure. If actually more than one is disclosed, for convenience only, such embodiments of the inventive subject matter may be referred to herein individually or collectively by the term "inventive," and are not intended to voluntarily limit the scope of this application to any single invention or inventive concept. Thus, while specific embodiments have been illustrated and described herein, any arrangement contemplated to achieve the same purpose may substitute for the specific embodiments shown. This disclosure is intended to cover any and all modifications or variations of the various embodiments. After reading the above description, combinations of the above embodiments, as well as other embodiments not specifically described herein, will be apparent to those skilled in the art.

Claims

1. A human-machine interface system, comprising: Multiple optical muscle motion imaging sensors, the optical muscle motion imaging sensors being configured to detect optical characteristic signals indicating tissue features and motion; A support structure configured to hold the sensor within an operable proximity range to the user's skin surface; The processor is programmed to analyze the optical characteristic signals, separate signal components related to specific physiological and anatomical features, and convert these components into user authentication and command inputs for interfacing with computing devices.

2. The system according to claim 1, wherein, Each optical motion imaging sensor is an optical sensor capable of quantifying changes in tissue optical properties for real-time user identification and intent interpretation, including but not limited to absorption, reflection, and density.

3. The system according to claim 1 or 2 further includes a machine learning module within the processor, the machine learning module being used to dynamically compare the input optical characteristic signal with a pre-established database including multiple user-specific signal distributions, thereby improving the accuracy of user identification and command interpretation.

4. The system according to claim 1, wherein, The processor is also configured to adaptively modify authentication parameters in response to detected changes in the user's physiological state, thereby maintaining a consistent and secure user interface experience.

5. The system according to claim 1, wherein, The optical kinesiology sensor is also configured as a reference sensor to calibrate the system based on the user's unique anatomy and movement patterns.

6. A method for real-time and continuous authentication in human-machine interaction, the method utilizing the system according to claim 1, and comprising: dOMG data is acquired via one or more optical myokinesis sensors; The optical characteristic signals are continuously processed for dynamic user authentication. Translate physiological or anatomical changes into specific control commands for interacting with digital devices.

7. The method according to claim 6, wherein, The continuous authentication process includes dynamic adjustments to specific physiological changes in the user, thereby ensuring personalized and secure interaction with the computing device.

8. The method according to claim 6, wherein, Translated control commands are used for secure transaction authentication in digital healthcare, medical, defense, or financial applications.

9. A method for operating a Mixed / Extended Reality (XR) interface, the method comprising: Use one or more optical motion sensors to detect user movements and gestures; Process detected motion and gestures to generate control signals; The control signals are used to manipulate one or more virtual elements within the XR environment.

10. A non-transitory computer-readable medium storing instructions that, when executed by a processor, perform the method according to claim 6 or 9.

11. Use of the system and / or method according to any one of claims 1-9 in an application, said application including but not limited to secure access control systems, personalized user interfaces, assistive technologies for individuals with physical disabilities, and interactive games or training environments.

Images