Tissue spectrophotometry for human-computer and human-machine interfaces

Abstract

Described herein are spectrophotometric methods and devices for determining the position and / or movement of a body part, such as a finger, hand, wrist, arm, etc. The apparatus and methods described herein use optical characteristics, such as one or more of absorption, transmission, and reflection, to accurately and quickly determine positions and / or motions that may be used to control one or more devices and / or as input to a computer or software.

Description

Priority declaration

[0001] This patent application claims priority to U.S. Provisional Patent Application No. 63 / 343,996, filed on May 19, 2022, entitled “TISSUE SPECTROPHOTOMETRY FOR HUMAN-COMPUTER AND HUMAN-MACHINE INTERFACING,” and U.S. Provisional Patent Application No. 63 / 424,844, filed on November 11, 2022, entitled “DYNAMIC BIO-SPECTROSCOPY FOR CONTINUOUS AND DISCRETE INPUT AUTHENTICATION,” each of which is incorporated herein by reference in its entirety. Incorporated by Reference

[0002] All publications and patent applications mentioned in this specification are herein incorporated by reference in their entirety to the same extent as if each individual publication or patent application was specifically and individually indicated to be incorporated by reference. background

[0003] Non-invasive interface systems that sense neuromuscular activation typically detect electrical signals generated by nerves, for example using electromyographic sensors, or detect movement or contraction of muscles, for example using force sensors such as myographic sensors. The output of these sensing systems can be provided as input to a computer system and / or actuator, such as a prosthetic device or robotic tool. Therefore, these systems are often equivalently referred to as human-machine interface systems, brain-computer interfaces (BCIs), brain-machine interfaces (BMIs), mind-machine interfaces (MMIs), or direct neural interface (DNI) systems.

[0004] However, currently available methods and devices for non-invasively reading neuromuscular signals have proven less than satisfactory to date. For example, surface-level electromyography (sEMG) in particular has a number of significant disadvantages, including motion artifacts, degradation of the electrical signal over time, and skin electrode sensitivity. What is needed is a sensing system and device that is robust, inexpensive, and insensitive to direct skin contact. This article describes methods and devices that can address these issues. Public Overview

[0005] Described herein are spectrophotometric (also referred to herein as optical myography or OMG) methods and apparatus for determining the position and / or motion of one or more body regions (and / or in some examples, the forces exerted by the body regions). In general, these methods and apparatus can non-invasively detect changes in optical property signals from tissue, such as changes in one or more of light absorption, light reflection, light density, etc. Signals generated by optical properties of tissue or changes in optical properties of tissue may be referred to herein as "optical property signals." These methods and apparatus can form a spectrophotometric representation consisting of multiple optical property signals on a skin region separate from the body region whose position and / or motion is being determined. The optical property signals can be processed to separate (e.g., remove) heartbeat-related or other non-specific signals from the optical property signals, and the resulting processed signals can be used to determine the position and / or motion of the body region. Thus, the optical property signal, and in particular the spectrophotometric representation consisting of a plurality of optical property signals (e.g., obtained on the surface of the skin), can be used directly as a control signal to control the operation of a device (machine, computer, software, etc.), or the optical property signal and / or the spectrophotometric representation comprising a plurality of optical property signals can be further processed and / or analyzed to explicitly infer a specific discrete or continuous body state (e.g., position) or motion in real time. As described below, any of these methods and apparatuses can also or alternatively determine a force exerted by a body region.

[0006] In general, these methods can be used to determine the position and / or motion (and / or in some cases, the forces exerted by one or more body parts (e.g., arms, hands, fingers, head, legs, feet, etc.) of one or more body parts by monitoring spectrophotometric representations from areas of skin separate from the body part whose position or motion is being determined. Areas of skin (e.g., "skin regions") can be located distal to the body parts; for example, the position and / or motion of a subject's fingers and hand can be accurately determined over time by monitoring spectrophotometric representations of skin regions on the subject's wrist, forearm, or back of the subject's hand.

[0007] Biomedical spectrophotometry uses reflected or transmitted light at different wavelengths to detect local changes in tissue absorption, emission, and / or reflection of light. The most common use of tissue spectrophotometry today is for non-invasive measurements of cardiac physiology, known as "pulse oximetry". Optical pulse oximeters typically operate in two ways: calculating blood oxygen saturation via differences in absorption at two wavelengths (usually red / IR, about 600nm to 900nm), and tracking changes in blood flow and / or small deformations of vascular structure via wavelengths of light at peaks in the absorption spectrum of venous blood (e.g., typically green, about 520nm) - sometimes also called photoplethysmography (PPG). These methods are widely used but have a common failure mode: voluntary muscle movement causes deformation of local tissue (relative to rigid bone), and muscle activity can cause changes in blood oxygen levels, which together cause changes in the optical properties of the tissue detected by spectrophotometry. The signal generated by muscle movement is very large and is independent of heart rate. Second, tissue is inherently heterogeneous and includes skin ridges, scar tissue, complex vascular structures, tendons, and other components, which result in uneven pathways for measuring oxygenated blood. In summary, both voluntary muscle movement and tissue heterogeneity are considered artifacts and present significant challenges to pulse oximetry technology.

[0008] Methods and devices are described herein that use spectrophotometric techniques to exploit the inherent heterogeneity of tissue and its dynamic changes over time to rapidly, accurately and non-invasively detect intended and unintended muscle movements in real time. These techniques are referred to herein as optical myography.

[0009] Using multiple spectrophotometric sensors that can be arranged in an array with a predetermined density and / or coverage area, changes in tissue optical properties (e.g., reflectance / absorbance / transmittance relative to a fixed position) over a coverage area generated by voluntary and involuntary movement can be accurately decoded to infer specific muscle states. Optically decoded muscle activity can be used as control signals for intuitive interaction and interfacing with computers and machines by leveraging a person's natural / native motor control experience.

[0010] As used herein, spectrophotometric decoding of states / transitions of body parts (e.g., muscles, fingers, hands, arms, etc.) may be broadly referred to as "optical myography" (OMG) and may serve as the basis for novel brain-machine interfaces.

[0011] For example, a wearable system is described herein, including: a group of spectrophotometric sensors configured to detect optical property signals from tissue; a support configured to hold an array of spectrophotometric sensors adjacent to a skin surface; and a processor configured to receive optical property signals (including a spectrophotometric representation of a sensed region) from the array of spectrophotometric sensors, separate (e.g., remove) a component of the optical property signal corresponding to a heartbeat from the received optical property signal, and determine the position and / or movement of a body region (e.g., a muscle or a body part including a muscle) from the spectrophotometric representation. The sensor responsible for receiving the optical property signal can also be used as a reference sensor, which can be used to align the spectrophotometric representation (e.g., adjust the alignment of the spectrophotometric representation), for example, based on determining the underlying anatomical structure from the spectrophotometric representation. In some examples, the optical property signal can be analyzed by the system (including in some examples by the group of spectrophotometric sensors).

[0012] As used herein, a spectrophotometric sensor group may include a plurality of spectrophotometric sensors and / or one or more light sources. The spectrophotometric sensor group may be configured as an array of sensors that may be configured to interrogate a predetermined area of ​​tissue (e.g., skin), such as 1 cm 2 or larger (e.g., 2 cm 2 or larger, 3cm 2 or larger, 4cm 2 or larger, 5cm 2 or larger, 10cm 2 or larger, 12.5cm 2 or larger, 15cm 2 or larger, 17.5cm 2 or larger, 20cm 2 or larger, 25cm 2 or larger, 30cm 2 or greater, etc.). The density of sensors and / or emitters (light sources) in a spectrophotometric sensor group can be configured to provide continuous or near-continuous coverage of tissue (e.g., with a gap between adjacent interrogation zones of each sensor of less than 0.1 mm, less than 0.2 mm, less than 0.3 mm, less than 0.5 mm, less than 0.6 mm, less than 0.7 mm, less than 0.8 mm, less than 0.9 mm, less than 1 mm, etc.). In some examples, the spectrophotometric sensor group can be configured so that interrogation zones of adjacent sensors overlap.

[0013] Any of these devices (e.g., systems, devices, etc., including hardware, software, and / or firmware) can be configured as a wearable device for detecting neuromuscular activity. Neuromuscular activity can include the position and / or movement of a region of a body part. Typically, the spectrophotometric sensor group can be configured to detect optical properties from an area of ​​tissue, and can include, for example, one or more light emitters and one (or more preferably multiple) optical detectors. The light emitter can be any suitable light emitter, such as, but not limited to, an LED, a laser, and the like. The light emitter can emit a single wavelength or color, a certain wavelength range, or a plurality of different discrete wavelengths (or discrete wavelength bands). For example, the light emitter can include an LED configured to emit red light. In some examples, the light emitter can be configured to emit light in the infrared (including near infrared) spectrum, such as light between about 700nm and about 800nm. In some examples, the light emitter can be configured to emit light between about 600nm and about 990nm. In some examples, light can be emitted continuously or in a pulsed manner (e.g., at a frequency between about 5 Hz and 1000 Hz, greater than 10 Hz, greater than 100 Hz, etc.) The light emitters can include multiple light emitters configured to emit at two or more different wavelengths or wavelength ranges.

[0014] The optical detector may be any suitable optical detector such as, but not limited to, a photodetector / photosensor, for example, a photodiode, a charge coupled device (CCD), a phototransistor, a quantum dot photoconductor, a photovoltaic device, a photochemical receptor, or the like.

[0015] Typically, the spectrophotometric sensor group can be integrated so that one or more light emitters are paired with one or more light sensors. In some examples, the spectrophotometric sensor group can include a single light emitter or a pair of light emitters that provide light to multiple light sensors. In some examples, one or more light emitters can be separated from one or more light sensors. The light sensors (equivalently referred to as spectrophotometric sensors in this article) can be arranged in an array (e.g., a 2D array). The emitters can be included in the array.

[0016] One or more light emitters and one or more light sensors (eg, an array of spectrophotometric sensors) may be arranged and / or secured to a support such that light emitters and light sensors of a spectrophotometric sensor group are arranged adjacent to each other and / or opposite to each other.

[0017] The methods and devices described herein can generally detect optical properties from a region of tissue, which can be related to the position and / or movement of a body part and / or the force applied by the body part, which is separate from the region of tissue being sensed by the spectrophotometric sensor. For example, as will be described herein, the optical property can be the absorption, emission and / or reflection of light by the tissue. These properties can form a spectrophotometric representation of a region of tissue. The spectrophotometric representation is similar to an image of a tissue (e.g., skin) and can be referred to as a spectrophotometric image in this article. Therefore, the spectrophotometric representation (image) can represent the spectrophotometric properties on a tissue region (such as the surface of the skin) and can be acquired over time to represent a spectrophotometric "video" of the tissue (e.g., skin). As described above, the spectrophotometric representation of the tissue provides dynamic information indicating the position and / or movement of nearby body parts (fingers, hands, arms, etc.), as well as reference or landmark information from relatively unchanging regions (e.g., moles, vascular systems, etc.). Thus, of surprising benefit, the methods and apparatus described herein can use spectrophotometric representations obtained by a group (e.g., an array) of spectrophotometric sensors for determining position and / or motion, and for registering the spectrophotometric representations as the group (e.g., an array of spectrophotometric sensors) moves relative to tissue and / or as the tissue changes (compresses, stretches, wrinkles, etc.) with movement of the subject.

[0018] In general, the methods and devices described herein can use detected optical properties (which can be part of a spectrophotometric representation of a tissue region) to determine position and / or motion by correlating the optical properties on a region of tissue (e.g., a spectrophotometric representation) with optical properties of the same (or nearly the same) region of tissue associated with a known position and / or motion of a body region. The correlation can be performed by comparing the data of the position and / or motion associated with the body region and the spectrophotometric representation, and / or using a statistical model based on previous spectrophotometric representations and the associated position or motion of the body region, and / or using a trained machine learning agent (e.g., a neural network). Any of these devices and methods can output the determined position and / or motion, or can output an indicator of the determined position and / or motion, such as, but not limited to, coordinates, vectors, models, etc. of one or more parts (joints, extremities, landmarks, etc. on a body part).

[0019] As described above, the method or device may include one or more processors, such as a microprocessor and / or additional circuits, which may be part of the control circuit. One or more processors may include instructions for executing any of the methods described herein. In particular, the processor may be configured to separate a component of the optical characteristic signal corresponding to the heartbeat to separate an optical signal generated by autonomous or involuntary muscle movement. For example, the optical characteristic may correspond to the differential absorption of light of two (or more) wavelengths. As will be described in more detail herein, in any of these methods and devices, one or more processors may also adjust the registration of the spectrophotometric representation (or an array of spectrophotometric sensors). One or more processors may determine the position and / or movement of a body part locally based on the spectrophotometric representation, or it may transmit the spectrophotometric representation for further remote processing to determine the position and / or movement of a body part.

[0020] Typically, the group of spectrophotometric sensors is fixed to a support. For example, the device may include a plurality of spectrophotometric sensors fixed to a support. The spectrophotometric sensors may be fixed to the support so that the relative positions between the spectrophotometric sensors are constant or fixed. In some examples, position shifts between the spectrophotometric sensors may be allowed, and the method or device may correct for small deviations between the positions of the spectrophotometric sensors during registration.

[0021] The support may be any structure configured to hold the spectrophotometric sensor adjacent to the tissue from which the signal will be measured. For example, the support may be or may include clothing, jewelry, or other wearable structures. For example, the support may be or may include a strap, a band, a patch, or a belt. The support may include an adhesive to hold the spectrophotometric sensor in an appropriate position relative to the skin region. The support may be configured to be fixed to the body (and in some cases removably fixed to the body). In some examples, the support is configured to be mounted on one or more of the following: the user's arm (e.g., forearm, shoulder, upper arm, wrist, elbow, hand and / or ringer, etc.), head (forehead, jaw, etc.), neck, torso (e.g., back, upper back, lower back, abdomen, etc.), leg (groin, thigh, knee, calf, ankle, foot, toe, etc.). The spectrophotometric sensor group may be coupled to the support. For example, the spectrophotometric sensor group may be rigidly coupled to the support and / or flexibly coupled to the support. In addition to the spectrophotometric sensor, the support may hold a processor (e.g., a controller, control circuitry, microprocessor, communication circuitry, memory, etc.) and / or a power source (e.g., a battery, a capacitive power source, a regenerative power source, etc.) and / or connections (e.g., wires, traces, etc.), etc. The support may include one or more housings surrounding all or part of the processor, power source, and / or spectrophotometric sensor set.

[0022] Any of the devices (apparatus, systems, etc.) described herein may also include one or more signal conditioners configured to modify (e.g., adjust) a signal of or from a spectrophotometric sensor group. For example, the signal conditioner may include one or more of a lens, a diffuser, a filter, and a lens array. The signal conditioner may be part of or separate from the spectrophotometric sensor group. The signal conditioner may be coupled to a support and / or at least partially enclosed within a housing. In some examples, the adjustment is part of the processor.

[0023] Typically, the processor can be configured to output an indicator of muscle movement. For example, the processor can be configured to wirelessly output an indicator of muscle movement. The processor can be configured to correlate the spectrophotometric sensor to one or more of muscle movement or body part movement. In some examples, the processor is configured to modify the device input based on the detection of activation of one or more muscles.

[0024] In any of these examples, the processor may include processing circuitry. As described above, the processor may include one or more dedicated microprocessors. In some examples, the processor is part of the device (e.g., coupled to a support). In some examples, the processor is a remote processor or is part of a remote processor. For example, processing of a signal (e.g., a spectrophotometric representation) may occur partially or completely locally (on the wearable portion), or processing may occur partially or completely remotely, such as using a remote processor. Processing output is typically done in real time, but in some cases, the signal (and / or output) may be stored for later viewing, analysis, and use, including for further training of the device, which will be described in more detail below.

[0025] For example, a wearable device (e.g., a system) for detecting the position (and / or movement) of a body part of a subject is described herein. The system may include: a plurality of spectrophotometric sensors configured to sense optical properties, wherein the plurality of spectrophotometric sensors include at least one light emitter and a plurality of optical detectors; a support configured to hold the plurality of optical detectors of the spectrophotometric sensors adjacent to a skin surface such that the plurality of optical detectors are arranged in a pattern relative to the skin surface; and a processor configured to receive an optical property signal from each of the plurality of optical detectors, separate a signal corresponding to a heartbeat from the optical property signal, and distinguish muscle movement corresponding to one or more muscles based on the received optical property signal. In addition, the device may be configured to register a spectrophotometric representation and / or device (e.g., a spectrophotometric sensor) by using a spectrophotometer sensor as a reference sensor to accurately determine the underlying anatomical structure from the spectrophotometric representation.

[0026] In any of these devices or methods, the processor can be configured to distinguish voluntary muscle movement from involuntary muscle movement from a received optical characteristic signal (e.g., a spectrophotometric representation). The processor can be configured to separate the optical signal corresponding to the heartbeat from the received optical characteristic signal by subtracting a periodic signal common to the plurality of optical detectors when generating the spectrophotometric representation.

[0027] The at least one light emitter may include, for example, an LED, and the plurality of optical detectors may include photodetectors. As described above, the apparatus may include a signal conditioner configured to modify the optical characteristic signal, which may include one or more of a lens, a diffuser, a filter, and a lens array.

[0028] In some examples, the support includes clothing, jewelry, etc. For example, the support can be a strap, a hoop, a patch, or a belt. The support can be configured to fit on one or more of a user's forearm, wrist, or hand, for example.

[0029] In any of the devices described herein, the support may be configured to maintain one (or more) spectrophotometric sensor groups in a relatively fixed arrangement relative to the skin surface and / or relative to other spectrophotometric sensor groups when worn by a user.

[0030] As described above, the processor can be configured to output the position and / or motion (or an indicator of the position and / or motion), and / or can be configured to wirelessly output the position / motion and / or the indicator of the position / motion. The processor can be configured to correlate one or more of the plurality of spectrophotometric sensors (e.g., a subset of the plurality of spectrophotometric sensors) with one or more of muscle movement or body part movement.

[0031] Methods of using any of these devices are also described herein, including methods of detecting movement and / or position of a body part (or a portion of a body part). For example, the method may include: non-invasively positioning a spectrophotometric sensor group on a muscle or tendon; detecting an optical characteristic signal from the muscle using the spectrophotometric sensor group; removing a component of the optical characteristic signal corresponding to a heartbeat from the optical characteristic signal; and outputting an indicator of muscle movement based on the optical characteristic signal.

[0032] Any of these methods may include determining whether the optical characteristic signal indicates a voluntary muscle movement. For example, determining whether the optical characteristic signal indicates a voluntary muscle movement may include using a trained neural network to determine whether the optical characteristic signal corresponds to a voluntary muscle movement.

[0033] In any of these methods, non-invasively positioning the spectrophotometric sensor group can include positioning the spectrophotometric sensor group on a proximal muscle or tendon to detect movement at a distal location. In some examples, non-invasively positioning the spectrophotometric sensor group includes positioning the spectrophotometric sensor group on a forearm to detect one or more of finger movement and position. For example, non-invasively positioning the spectrophotometric sensor group can include wearing one or more of a garment, a band, a hoop, a belt, or a patch. Detecting an optical property signal from a muscle using the spectrophotometric sensor group includes emitting one or more wavelengths of light from an emitter of the spectrophotometric sensor group, and detecting optical properties of the one or more wavelengths of light using one or more optical detectors of the spectrophotometric sensor group.

[0034] Typically, the method may include outputting an indicator of muscle movement based on the optical characteristic signal (e.g., indicating a neural command for muscle movement). The output may be a signal (a signal for presentation (e.g., display), a signal for recording, and / or a signal for further processing) and / or the output may be used to control actuation of a device. The device may be a mechanical and / or electrical device (e.g., a prosthetic device, a robotic device, etc.) or any combination thereof. Any suitable output device may be used, including devices that would otherwise be controlled by the user's muscle movement, including devices that can be turned on / off or adjusted. In some examples, the output may be provided to software, for example, to control a software avatar, etc.

[0035] Any of these methods may include outputting an indicator of neural activity and / or muscle movement based on an optical characteristic signal that indicates movement of a muscle or body part. For example, outputting an indicator of muscle movement may include triggering an effector based on the muscle movement. Outputting an indicator of muscle movement may include transmitting the indicator of muscle movement to a remote processor.

[0036] Any of these methods may include identifying a correspondence between a spectrophotometric sensor group and a specific anatomical location. For example, the optical characteristic signal also embodies a reference identifier that can be used to identify a correspondence between the spectrophotometric sensor group and muscles and / or movements. Any of these methods may include the reference identifier that can be used when the device is repositioned. For example, when a device equipped with a spectrophotometric sensor is used intermittently, the sensor position may not be exactly the same as the previous position, which may adversely affect the model that relates the optical characteristic signal to the movement. The reference marker can be used to reconstruct the previously established correlation between the optical characteristic signal and the muscle movement. Therefore, a previously trained model (e.g., a neural network) can be used to associate muscle movement based on the optical characteristic signal at a subsequent time.

[0037] A method of detecting muscle movement may include: non-invasively positioning a spectrophotometric sensor set on a muscle or tendon; detecting an optical characteristic signal from the spectrophotometric sensor set; processing the optical characteristic signal to separate a global optical characteristic signal from the detected optical characteristic signal; determining whether the processed optical characteristic signal indicates voluntary or involuntary muscle movement; and outputting an indicator of voluntary muscle movement based on the processed optical characteristic signal. Another step of using a fiducial marker to verify whether this is a location similar to a previously observed location may be performed.

[0038] For example, this document describes a method for determining the position and / or movement of a region of a subject's body by: obtaining a spectrophotometric representation of a region of the subject's skin by collecting data from each photometric sensor in an array of spectrophotometric sensors; adjusting the alignment of the spectrophotometric representation by comparing the spectrophotometric representation of the region of the subject's skin with a previous spectrophotometric representation of the region of the subject's skin to form an adjusted spectrophotometric representation of the region of the subject's skin; and determining the position and / or movement of the region of the subject's body using the adjusted spectrophotometric representation of the region of the subject's skin.

[0039] For example, methods are described herein for determining the position and / or movement of a region of a subject's body by collecting data from each spectrophotometric sensor in an array of spectrophotometric sensors to obtain a region of the subject's skin covering 1 cm 2 or a larger area of ​​a portion of the subject's skin; adjusting the alignment of the spectrophotometric representation by comparing the spectrophotometric representation to a previous spectrophotometric representation of the portion of the subject's skin to account for one or more of: movement of the array of spectrophotometric sensors relative to the subject's skin or changes in the shape of the subject's skin; and determining and outputting the position and / or movement of the portion of the subject's body using the adjusted spectrophotometric representation of the portion of the subject's skin.

[0040] Any of these methods may include outputting the position and / or motion of a portion of a subject's body (e.g., a finger, hand, wrist, arm, etc.). The output position / motion (or an indicator of the position / motion) may be used to operate one or more devices based on the determined position / motion of the portion of the subject's body. In some examples, the determined position and / or motion may be used as an input to one or more of a control computer and / or software or firmware.

[0041] Any of these methods may include the step of repeatedly acquiring a spectrophotometric representation to determine movement of a region of a subject's body, and / or the step of repeatedly adjusting a registration of the representation of the spectrophotometric representation. The registration step (e.g., the step of adjusting the registration of the spectrophotometric representation) may be performed each time the spectrophotometric representation is acquired, or may be performed less frequently. For example, the step of registering the spectrophotometric representation may be performed every two times, every five times, every ten times, every 30 times, every 50 times, every 60 times, every 100 times, etc., when the spectrophotometric representation is acquired.

[0042] The method may include iterating to continuously or nearly continuously determine the position and / or movement of a region of the body (e.g., a body part such as one or more fingers, hands, wrists, arms, head, legs, etc.). For example, any of these methods or devices may be configured to repeat the step of acquiring the spectrophotometric representation at a frequency of 5 Hz or more (10 Hz or more, 15 Hz or more, 20 Hz or more, 30 Hz or more, 60 Hz or more, etc.). As described above, the step of registering the spectrophotometric representation may be performed at the same or lower rate as the rate at which the spectrophotometric representation is acquired (e.g., 0.1 Hz or more, 0.2 Hz or more, 0.5 Hz or more, 1 Hz or more, 2 Hz or more, 5 Hz or more, etc.).

[0043] Any of the apparatus and methods described herein may be configured to operate on a collection of spectrophotometric representations (eg, a video or a spectrophotometric video) rather than on discrete spectrophotometric representations ("spectrophotometric images").

[0044] As described above, determining the position and / or movement of the region of the subject's body may include continuously determining the position of the region of the subject's body by repeating the steps of obtaining a spectrophotometric representation and adjusting a registration of the representation.

[0045] Obtaining a spectrophotometric representation of a region of the subject's skin may include obtaining a spectrophotometric representation of a region of any suitable size. For example, obtaining a spectrophotometric representation of a region of the subject's skin may include obtaining a region of the subject's skin covering 1 cm 2 or larger area (e.g., 2 cm 2 or larger, 3cm 2 or larger, 4cm 2 or larger, 5cm 2 or larger, 10cm 2 or larger, 12.5cm 2 or larger, 15cm 2 or larger, 17.5cm 2 or larger, 20cm 2 or larger, 25cm 2 or larger, 30cm 2 or larger, etc.). The section may have any shape (e.g., rectangular, square, elliptical, etc.). In some cases, the section may be continuous or nearly continuous.

[0046] Adjusting the representation registration can include transforming the spectrophotometric representation using any suitable registration technique to account for movement of the array of spectrophotometric sensors relative to the subject's skin and / or to account for changes in the shape of the subject's skin. For example, any of these methods can use rigid or non-rigid transformation techniques (such as (but not limited to) linear transformations (e.g., rotations, scaling, translations, and other affine transformations) and elastic (non-rigid) transformations (such as but not limited to radial basis functions (e.g., thin plate or surface splines, multi-quadric surfaces, and compact support transformations)), physical continuum models, and large deformation models (e.g., diffeomorphisms).

[0047] As described above, any of these methods or devices may include using a trained machine learning agent to determine the position and / or movement of a region of a subject's body. The machine learning agent may be trained based on data or information from a subject on whom the method is performed, or it may be trained based on data or information from a separate one or more test subjects. For example, the method or device described herein may use a trained machine learning agent to determine the position and / or movement of a region of a subject's body, the trained machine learning agent having been trained using a training data set comprising a plurality of previous spectrophotometric representations of skin and a plurality of video representations, the plurality of previous spectrophotometric representations being obtained using a test array of spectrophotometric sensors, the plurality of video representations showing the region of the test subject's body at a time corresponding to each previous spectrophotometric representation from the plurality of previous spectrophotometric representations. Alternatively or additionally, determining the position and / or movement of a region of the subject's body may include using a machine learning agent that has been trained using a training dataset, the training dataset comprising a plurality of previous spectrophotometric representations of the subject's skin and a plurality of video representations, the plurality of previous spectrophotometric representations acquired using an array of spectrophotometric sensors, the plurality of video representations showing the region of the subject's body at a time corresponding to each previous spectrophotometric representation of the plurality of previous spectrophotometric representations.

[0048] Any of these methods and apparatuses may include training a machine learning agent based on a training data set comprising a plurality of spectrophotometric representations of a subject's skin obtained from an array of spectrophotometric sensors and corresponding video representations of regions of the subject's body. If a machine learning agent trained based on a separate or different one or more test subjects is used, the apparatus or method may, for example, calibrate the machine learning agent using a set of specified calibration movements. These calibration movements may be instructed or unprompted (e.g., using statistics of a person's natural movements to provide unsupervised or semi-supervised calibration).

[0049] Alternatively, in any of these methods and apparatus, the position and / or motion of a region of the subject's body may be determined using a statistical model that specifies a relationship between the spectral representation and the position and / or motion of a body region (e.g., a body part such as one or more fingers, a hand, etc.) Any suitable statistical modeling may be used, including parametric, non-parametric, and semi-parametric models, such as regression modeling (e.g., polynomial and linear regression, etc.), classification models, etc.

[0050] Any of these methods may include using a wearable device that holds an array of spectrophotometric sensors against a region of the skin to obtain a spectrophotometric representation.

[0051] Also described herein are systems for performing any of these methods. For example, a system for determining the position and / or movement of a portion of a subject's body may include: a spectrophotometric sensor group comprising an array of spectrophotometric sensors configured to obtain a spectrophotometric representation of a portion of the subject's skin; a support member configured to hold the spectrophotometric sensor group adjacent to a surface portion of the subject's skin; a control circuit comprising one or more processors; and a memory coupled to the one or more processors, the memory storing computer program instructions that, when executed by the one or more processors, are configured to iteratively: obtain a spectrophotometric representation of the portion of the subject's skin by collecting data from each photometric sensor in the array of spectrophotometric sensors; adjust the alignment of the spectrophotometric representation by comparing the spectrophotometric representation of the portion of the subject's skin with a previous spectrophotometric representation of the portion of the subject's skin to form an adjusted spectrophotometric representation of the portion of the subject's skin; and determine and output the position and / or movement of the portion of the subject's body using the adjusted spectrophotometric representation of the portion of the subject's skin.

[0052] As described above, the program instructions, when executed by one or more processors, may be configured to iteratively acquire the spectrophotometric representation at a frequency of 5 Hz or higher and adjust the representation registration.The program instructions may be configured to separate out a component of the optical characteristic signal corresponding to a heartbeat.

[0053] As described above, the spectrophotometric sensor group may include a light emitter (e.g., a photodiode, LED, laser, etc.) and an optical detector (e.g., a photodetector, CMOS, CCD, etc.). Any of these devices may include a signal conditioner comprising one or more of a lens, a diffuser, a filter, and a lens array, the signal conditioner being configured to modify the spectrophotometric representation of a region of the subject's skin. The support may include a garment, a strap, a hoop, a patch, or a belt, etc. In some examples, the support is configured to fit on one or more of a forearm, a wrist, or a hand of a user. The processor may be configured to wirelessly output the position and / or movement of a region of the subject's body using the adjusted spectrophotometric representation of the region of the subject's skin. The processor may include a remote processor.

[0054] As described above, the device (eg, an array of spectrophotometric sensors) may be configured so that the spectrophotometric representation of the region may cover 1 cm of the subject's skin. 2 or larger area. One or more processors may be configured to adjust the representation alignment using rigid or non-rigid transformation techniques to account for movement of the array of spectrophotometric sensors relative to the subject's skin and / or to account for changes in the shape of the subject's skin. In some examples, the processor is configured to adjust the representation alignment using a non-rigid transformation. For example, the processor may be configured to adjust the representation using an affine transformation. The processor may be configured to determine and output the position and / or movement of a region of the subject's body using a machine learning agent that has been trained using a training data set that includes multiple previous spectrophotometric representations of the skin and multiple video representations, the multiple previous spectrophotometric representations being obtained using a test array of spectrophotometric sensors, the multiple video representations showing the region of the test subject's body at a time corresponding to each previous spectrophotometric representation in the multiple previous spectrophotometric representations.

[0055] For example, a system for detecting neuromuscular activity may include: a spectrophotometric sensor group, the spectrophotometric sensor group including an array of spectrophotometric sensors, wherein each spectrophotometric sensor is configured to detect an optical characteristic signal of a spectrophotometric representation corresponding to a region of a subject's skin from tissue; a support member, the support member configured to maintain the spectrophotometric sensor adjacent to the skin surface; and a processor, the processor configured to receive the spectrophotometric representation from the spectrophotometric sensor group and, after separating a component of the optical characteristic signal corresponding to a heartbeat from the received optical characteristic signal, detect muscle movement from the spectrophotometric representation.

[0056] The processor may be configured to adjust the registration of the spectrophotometric representation by comparing the spectrophotometric representation of the portion of the subject's skin to a previous spectrophotometric representation of the portion of the subject's skin to form an adjusted spectrophotometric representation of the portion of the subject's skin.

[0057] The processor may be configured to separate a component of the optical characteristic signal corresponding to a heartbeat to detect muscle movement.

[0058] The present invention also describes a wearable system for detecting neuromuscular activity, the system comprising: a plurality of spectrophotometric sensors configured to sense optical properties, wherein the plurality of spectrophotometric sensors include at least one light emitter and a plurality of optical detectors, wherein each spectrophotometric sensor is configured to detect an optical property signal corresponding to a spectrophotometric representation of a region of a subject's skin from tissue; a support configured to hold the plurality of optical detectors of the spectrophotometric sensor adjacent to a skin surface such that the plurality of optical detectors are arranged in a pattern relative to the skin surface; and a processor configured to receive an optical property signal from each of the plurality of optical detectors, separate a signal corresponding to a heartbeat from the optical property signal, and distinguish muscle movements corresponding to one or more muscles based on the received optical property signal.

[0059] A method of detecting the position and / or movement of a body region may include positioning a spectrophotometric sensor group on a skin region on a subject's body, wherein the skin region is not part of the body region; collecting a spectrophotometric representation including a plurality of optical characteristic signals from the skin region using the spectrophotometric sensor group; modifying the plurality of optical characteristic signals by separating one or more components of the optical characteristic signals corresponding to a heartbeat from the optical characteristic signals to form a modified spectrophotometric representation of the skin region; and outputting an indicator of the position and / or movement of the body region based on the modified spectrophotometric representation. The method may include transforming the modified spectrophotometric representation to account for one or more of: movement of the spectrophotometric sensor group relative to the skin region and shape changes in the shape of the skin region. For example, the transformation may include registering the modified spectrophotometric representation by comparing the modified spectrophotometric representation of the skin region with a previous spectrophotometric representation of the skin region, and transforming the modified spectrophotometric representation based on the comparison. Any of these methods may include determining the position and / or motion of a body region by correlating a spectrophotometric representation with a previous position and / or motion of the body region. Correlating the spectrophotometric representation with the previous position and / or motion of the body region may include using a machine learning agent that was trained using multiple previous spectrophotometric representations of the skin and multiple video representations that show the region of the test subject's body at a time corresponding to each previous spectrophotometric representation from the multiple previous spectrophotometric representations. In some examples, correlating the spectrophotometric representation with the previous position and / or motion of the body region includes using a machine learning agent that was trained using multiple previous spectrophotometric representations of the skin region and multiple video representations that show the region of the body at a time corresponding to each previous spectrophotometric representation from the multiple previous spectrophotometric representations.

[0060] A system for determining the position and / or movement of a body region may include: a spectrophotometric sensor group, the spectrophotometric sensor group including an array of spectrophotometric sensors, wherein each spectrophotometric sensor is configured to detect an optical property signal from tissue, wherein the detected optical property signal corresponds to a spectrophotometric representation; a support configured to hold the spectrophotometric sensor adjacent to a skin surface; and a processor configured to: receive the spectrophotometric representation from the spectrophotometric sensor group, adjust the registration of the spectrophotometric representation by transforming the spectrophotometric representation based on a comparison between the spectrophotometric representation and a previous spectrophotometric representation to form an adjusted spectrophotometric representation of a region of the subject's skin, and detect the position and / or movement of the body region based on the adjusted spectrophotometric representation after separating a component of the optical property signal corresponding to a heartbeat from the received optical property signal. Anatomical features (e.g., "landmarks") may include skin ridges, vascular structures, tendons, or any combination thereof.

[0061] All methods and apparatus described herein (in any combination) are contemplated herein and may be used to achieve the benefits described herein. BRIEF DESCRIPTION OF THE DRAWINGS

[0062] A better understanding of the features and advantages of the methods and apparatus described herein will be obtained by referring to the following detailed description setting forth illustrative embodiments and the accompanying drawings, in which: Figure 1 is an example of a schematic diagram of a wearable system for detecting neuromuscular activity as described herein.

[0063] Figure 2 One example of positioning a set of photometric sensors on a forearm to detect fiducial features and neuromuscular activity as described herein is schematically illustrated.

[0064] Figure 3A An example of a wearable system configured as a strip for detecting changes in optical properties of tissue due to neuromuscular activity is schematically shown.

[0065] Figure 3B Another example of a wearable system for detecting neuromuscular activity configured as a band is schematically shown.

[0066] Figure 3C Another example of a wearable system for detecting neuromuscular activity configured as a band is schematically shown.

[0067] Figure 4 A method of processing spectrophotometric sensor group data as described herein is shown.

[0068] Figure 5shows that when a user taps the ring finger ten times and then repeats a cycle of tapping only the index finger ten times, the amount of data received from a device worn on the forearm as described herein (similar to Figure 2 An example of spectrophotometric sensor group data obtained using the device shown in ).

[0069] Fig. 6A Shows something like Figure 5 An example of the spectrophotometric sensor set data (bottom) shown in , where the concurrent electromyogram (EMG) recording (top) shows five repeated tappings at about 1 Hz with the ring finger of the left hand.

[0070] Figure 6B The images are shown in Figure 2. The images from four spectrophotometric sensor groups (e.g., Figure 2 overlay of data of the device being worn).

[0071] Figure 6C Shown is a comparison of concurrent electromyography (EMG) and optical myography (OMG) recordings aligned with grasping movements, presented as raster plots.

[0072] Fig.6D Example images from real-time reconstruction of hand and finger positions from OMG signals compared to ground truth are shown.

[0073] FIG. 6E to FIG. 6E 4 shows example performance metrics for real-time reconstruction of hand and finger positions compared to real data.

[0074] Fig. 6F The process of using fiducial markers to align and / or register OMG data over time and / or across sessions is schematically illustrated.

[0075] Figure 6G Decoding performance before and after registration / alignment as described herein is shown.

[0076] Figure 7 One example of a method of using one or more spectrophotometric sensor sets as described herein to detect neuromuscular and fiducial markers to align and / or register OMG data over time and / or across sessions is schematically illustrated.

[0077] Figure 8 One example of a method of detecting neuromuscular activity using one or more spectrophotometric sensor groups as described herein is schematically illustrated.

[0078] Fig. 9An example of a method of determining the position and / or movement of a region of a body is schematically illustrated. Detailed Description

[0079] Described herein are spectrophotometric methods and apparatus for detecting neuromuscular activity, such as the position and / or movement of a region of a subject's body and / or forces exerted by a region of a subject's body. Typically, these apparatus and methods use optical properties, such as one or more of absorption, transmission, and reflection that occur as a result of neural signaling that causes muscle movement, which can cause changes in blood oxygenation, changes in blood flow, and / or changes in vascular structure (via deformation). These methods can generate a spectrophotometric representation of an area of ​​a user's skin, as well as adjust the spectrophotometric representation, including in some cases registering the spectrophotometric representation, and can determine the position and / or movement of a body part (e.g., a finger, hand, etc.) separate from the skin region and / or the force exerted by the body part.

[0080] Any of the methods and apparatus described herein can determine optical properties of an area of ​​tissue (e.g., skin) and separate the component of the sensed optical properties that corresponds to the heartbeat from the spectrophotometric signal, such that the resulting signal will primarily reflect signals generated by body position and movement that is independent of the heartbeat (e.g., which is caused by neuromuscular activity). In addition, the spectrophotometric representation can include underlying anatomical structures that can be used as fiducial landmarks for transforming the spectrophotometric representation, which provides significant accuracy and flexibility to the techniques and apparatus described herein. In general, the methods and apparatus described herein can detect or infer position and / or movement (e.g., neuromuscular activity) by detecting the optical properties of an area of ​​tissue.

[0081] As used herein, a "spectrophotometric representation" includes multiple optical property signals. For example, a spectrophotometric representation may include an array of optical property signals recorded from a tissue (e.g., a skin surface). The spectrophotometric representation may be configured as an array of optical property signals arranged spatially (e.g., corresponding to an area covered by a spectrophotometric sensor). In some examples, the spectrophotometric representation may include time information and may include changes in the optical property signals over time. The optical properties may be specific to one or more wavelengths of light, either individually or in combination. Thus, the spectrophotometric representation may map the optical response of the tissue in regions (space) and in some examples over time periods. Typically, the spectrophotometric representation may provide a comprehensive visualization of tissue regions from which potential physiological features may be determined. For example, the spectrophotometric representation may be used to accurately decode complex body movements by associating these movements with unique spectral changes induced near or far from body tissue. In practice, the spectrophotometric representation may be a data structure, an image, a video, or the like.

[0082] In one example, the device is comprised of a wearable structure ("wearable"), such as a cuff or band (e.g., a 3D printed band), that is used as a flexible support to hold multiple spectrophotometric sensors in communication with a skin region to detect spectrophotometric signals. Figure 1 An example of a spectrophotometric system or subsystem as described herein is shown. The spectrophotometric system can be used to perform optical myography. In this example, a wearable frame 105 can be configured as a band, hoop, garment, brace, patch, etc. that is configured to be worn adjacent to or against the skin of a subject. The system also includes one or more spectrophotometric sensor groups 101 as described herein, and processing circuitry 103 for processing signals from the spectrophotometric sensors. For example, the processing circuitry can include a flexible printed circuit board (PCB) or microwire interconnects, and a microprocessor that provides power and a common ground to the sensors. The microprocessor can also provide signal processing. Optionally, the processing can be performed directly by a processor 105 (e.g., a computer) without the need for intermediate circuits. Data recorded from the spectrophotometric sensors can be streamed to processors and other devices 107 for closed-loop control, such as Figure 1 Schematically shown in FIG.

[0083] exist Figure 1 In the embodiment of the present invention, the spectrophotometric system includes one or more spectrophotometric sensor groups having both an emitter and a sensor (e.g., an emitter / sensor including a pair or combination of an optical emitter and an optical sensor) that provide input to a processor. The processing circuit 103 and / or the processor 105 may include modules for processing signals to filter, amplify, and / or detect or determine muscle movement. In some examples, the processing circuit 103 may be a microprocessor. Optionally, the processing circuit 103 may communicate with a computer 105 or other (auxiliary) processor that may store, transmit (e.g., to a remote or local server), or process data from the spectrophotometric sensors.

[0084] The spectrophotometric sensor group can perform spectrophotometric measurements and can include a light source that emits at a specific wavelength, such as a photodiode. The light can be generated by an LED or a coherent light source (e.g., a laser). The light source can be time-pulsed for signal multiplexing or time-of-flight measurement, or for ambient light elimination. The spectrophotometric sensor can also include a light detector, such as a photodetector and / or a CMOS. The detector can have a filter that only accepts light of a specific wavelength. The spectrophotometric sensor can include one or more filters, lenses, diffusers, lens arrays coupled to either or both of the emitter and the detector.

[0085] The spectrophotometric sensor group 101 (including one or more light sources and one or more light detectors) can be held by a wearable support 106 (e.g., a frame or other structure) that holds the spectrophotometric sensor group relative to the skin and other spectrophotometric sensor groups so that optical measurements can be made from the skin. Unlike EMG signals that are sensitive to skin contact and changing skin conductivity, the spectrophotometric measurements described herein are robust and relatively insensitive to these variable characteristics and can remain stable over time and even between patients. The devices described herein can be used across different subjects regardless of skin color (e.g., skin color), age, etc. As will be described in more detail below, the devices and methods can adjust the position (which can shift between subjects and over time) to interpret the spectrophotometric signals, thereby determining neuromuscular activity, for example, by using adaptive components including machine learning components.

[0086] In some examples, the spectrophotometric sensors are arranged in a two-dimensional array outside the subject's skin, for example, with a clustered geometry relative to the target neuromuscular region (e.g., including but not limited to radial placement around the arm; above the muscle). The sensor may include surface mount components (e.g., photodiodes and photodetectors) that can be printed directly on a flexible circuit board or printed on a rigid board that can be flexibly linked together by flexible interconnects. The spectrophotometric sensor may include two main subcomponents: a light emitter and a light detector. The sensor may include many individual sensing units, such as a CMOS composed of multiple pixels. The number of sensing elements and / or emitting elements within the sensor and the number of sensors in the sensor array may have an arbitrarily high density. In some examples, the spectrophotometric sensing element includes a minimum of two sensors (three sensors, four sensors, five sensors, eight sensors, 10 sensors, 15 sensors, 16 sensors, etc.). An apparatus including a sensor and / or a processing circuit (e.g., a control circuit) may be configured to separate heartbeats using common mode rejection. Another example of a spectrophotometric sensor is a single light emitting photodiode and a single 480x640 pixel CMOS detector. Any suitable number and arrangement of spectrophotometric sensor groups can be used. For example, in some examples, 2 to 30 sensors are used (e.g., 2 to 26 sensors, 2 to 24 sensors, 4 to 30 sensors, 4 to 26 sensors, 4 to 24 sensors, 4 to 18 sensors, 8 to 30 sensors, 8 to 24 sensors, 8 to 16 sensors, etc. can be used).

[0087] The light applied by the spectrophotometric sensor group can be structured, coherent, or diffusely generated by an LED or coherent light source (e.g., a laser). The emitted light can be polarized, and a polarization filter can be used. The emitted light can be pulsed for "time of flight" measurements to allow volumetric tissue measurements / estimations (e.g., changes in tissue optical properties deep below the surface of the skin). Measurements of refracted or birefringent light can also be performed.

[0088] As described above, any signal conditioner may also be used, including lenses, diffusers, filters, lens arrays, etc. Alternatively or additionally, the skin itself may be used as a "lens" (e.g., structured / random diffuser) for optical phase reconstruction and thus for computational volume reconstruction. Different wavelengths (e.g., visible and IR light) may be used in proportion, which may help suppress noise.

[0089] Amplifiers can be used to maximize the dynamic range of the sensor signal to match the bit depth of the digitizing microprocessor. The microprocessor can be used to digitize the data from multiple sensors and send the data to a computer. The device can be configured to transmit optical data wirelessly. For example, the data can be streamed wirelessly, such as via Bluetooth, Wi-Fi, or RF (e.g., 2.4 GHz, 5.8 GHz, etc.). The data can also be streamed directly to a computer or used to directly control one or more devices using a wired interconnect.

[0090] As described above, in general, the methods and apparatus described herein can separate out and / or isolate a portion of the optical characteristic signal corresponding to a heartbeat, such as that which can be measured by a typical pulse oximeter measurement. Other pre-processing of the spectrophotometric representation (or the component optical signals that make up the spectrophotometric representation) can also be performed, including filtering, amplification, etc. In some examples, the apparatus can include a common average reference (mean subtraction of all signals) to do this. This can also separate signals from nearby action motion (e.g., movement of the torso or arm when measured from the wrist, hand, etc.).

[0091] In any of the methods and apparatus described herein, a spectrophotometric representation may be used to identify the position and / or motion of a body region, or an indicator of the position and / or motion of a body region. Typically, the methods and apparatus may identify the position and / or motion of a body region by correlating a current spectrophotometric representation with a previous spectrophotometric representation associated with the position and / or motion of the body region. Correlating the current spectrophotometric representation with the previous spectrophotometric representation (and the associated position and / or motion) may be performed by any suitable technique, including by using a trained machine learning agent and / or by statistical modeling, i.e., using one or more of Gradient Boosted Regression, Random Forest Regression, Bayesian, etc.

[0092] In any of the methods and apparatus described herein, spectrophotometric data (e.g., spectrophotometric representations) may be compressed or reduced in size in any suitable manner. For example, the size of a spectrophotometric representation obtained from any of these apparatuses and methods may be reduced by reducing the dimensionality of the data. The dimensionality reduction may be performed by principal component analysis (PCA), independent component analysis (ICA), or any linear or nonlinear eigendecomposition of the spectrophotometric representation. This may allow for the collection, storage, transmission, and / or processing of even large amounts of spectrophotometric data (e.g., spectrophotometric representations). For example, the size of the spectrophotometric representation may be reduced prior to creating and / or running a model, such as when training a machine learning agent.

[0093] calibration Any of the methods and devices described herein may include calibration. Typically, a calibration phase may be included to train the device to coordinate a particular spectrophotometric signal or set of signals with a particular neuromuscular activation. This calibration period may be performed during initial use for a particular user, and / or may be performed at the beginning of each application for the same user (or may be performed in an abbreviated form and / or alternative form), and / or may be performed periodically during operation. Calibration may be performed manually and / or semi-manually and / or automatically. Multiple different calibrations may be combined.

[0094] During the calibration period, the device can receive input from the user, can detect and / or can infer the position of the device worn on the body. In some examples, the device can determine the orientation of the device, and in particular the orientation of the spectrophotometric sensor group relative to the detected neuromuscular region. For example, the wearable device can be configured to be worn at a predetermined position on the body (e.g., arm, wrist, finger, elbow, hand, shoulder, upper arm, chest, neck, head, waist, torso, back, leg, knee, foot, etc.), and the position can be confirmed. In other cases, the camera can be used for pose estimation of a body part and correlating the pose estimation with an optical characteristic signal. The device can also confirm the relative orientation of one (or more) spectrophotometric sensor groups relative to a specific subject. The in vitro positions corresponding to different nerve and muscle regions (neuromuscular regions) may be different between different subjects. In addition, the position and / or orientation of the device may be slightly shifted for the same user during use and between multiple users. Therefore, calibration may be particularly helpful.

[0095] After the initial calibration, the device can be recalibrated across multiple sessions using the neuromuscular region. This may be necessary when the device is removed and then reattached, or if the device is repositioned on the wearer.

[0096] The methods and apparatus may use a processor (a local processor and / or a remote processor, such as Figure 1 The device may store an index (e.g., an index function, also referred to herein as a calibration index or simply an index) that matches the spectrophotometric sensing group with the user's neuromuscular activity / movement. The index may be updated as described above.

[0097] In any of these devices, the processor can generate an initial index using a machine learning agent that can be trained during an initial calibration period. In some examples, the device and / or method can instruct the user to perform a series of preset movements, and the spectrophotometric data output from the spectrophotometric sensor group can be recorded. For example, for a device worn on a forearm that can detect finger and / or wrist movements, the device or method can include instructing the user to move each finger (e.g., tap each finger each time, type a set phrase on a keyboard, etc.) while recording the spectrophotometric signal. The device and method can also instruct the user to move the forearm up / down, etc., in order to determine rough background motion. In one example, a keyboard or other input terminal can be used, which can receive keyboard input as well as spectrophotometric sensor data, and this information can be used to determine calibration.

[0098] Any of the methods and devices described herein may also use electrical stimulation for calibration and direct muscle innervation. In some examples, the system may include one or more electrodes for applying electrical energy to a nearby muscle (e.g., transcutaneous electrical energy electrodes, such as TENS) and measuring the effect on an optical signal measured by a set of spectrophotometric sensors associated with the same or nearby muscles. This may be performed during a calibration phase.

[0099] Typically, the calibration phase may include identifying or confirming the position of the device relative to the muscles and / or body regions of the user. For example, the methods and devices may include one or more additional calibration sensors or sensing modules, including electrical energy. In some examples, static vascular structures (larger veins) may be used as reference signals. In some examples, skin markers (e.g., skin wrinkles, hair, scars, birthmarks, time-varying neuromuscular activity itself, etc.) may be used as reference signals or markers for the device, particularly when orientation is performed on the same user between applications of the device.

[0100] Any of the devices described herein may also include one or more inertial measurement sensors, position sensors, or acceleration sensors, such as accelerometer / gyroscope sensors. These position sensors and / or motion sensors may be used to provide calibration signals as part of device calibration and / or for improving signal processing. For example, a user may move their body while wearing the device, and the device may match the motion with an optical property signal from the received spectrophotometric signal.

[0101] Any of these devices can use one or more cameras to track body motion so that they can be paired with the spectrophotometric signals detected by the worn system. For example, the device can use computer vision of the camera associated with the signal to train body reconstruction during and / or after the calibration phase. In some examples, the device can use regression based on IR dot matrix to train body reconstruction. In some examples, the device can train the motion / interface decoder based on expected or indicated actions. In some cases, the device can train the motion / interface decoder based on user feedback / input via self-report or indicated report. In other cases, the camera can use pose estimation of the body part and associate the pose estimation with the optical property signal. In some examples, the device can train the motion / interface decoder based on the assumed state or action (i.e., sleeping, dreaming).

[0102] In any of the methods and apparatus described herein, machine learning (ML) agents may be used, including but not limited to artificial intelligence trained networks. Alternatively or additionally, statistical methods may be used, such as, for example, regression, etc. This may be used for calibration as described herein to convert / correlate spectrophotometric signals to muscle (or in some examples, emotional) states and changes in these muscle / emotional states on millisecond to second time scales.

[0103] In addition to initial calibration, when a user first wears the device, the devices and methods may include performing an inter-use calibration any time the device is removed and then re-donned, such as a daily calibration. As used herein, calibration (and particularly inter-use calibration) may also be referred to herein as alignment. These methods and devices may also include in-use (e.g., during use) calibration, as the device may shift or move during operation. As described above, any bio-marker point may be used to help calibrate / align the device. For example, local static tissue features (such as vascular structures or skin features) may be used as alignment landmarks / reference signals.

[0104] Neuromuscular activity testing In general, the methods and apparatus described herein can use spectrophotometric data (including spectrophotometric representations, which may also be referred to as spectrophotometry and / or optical myography) to detect changes in tissue absorption / reflection of light at one or more (e.g., two) wavelengths to obtain information about internal states, movements, and intended actions of specific neuromuscular regions corresponding to specific movements. The spectrophotometric representations can be referred to as spectrophotometric images and can represent the optical properties of tissue over an area of ​​the body surface (e.g., over a skin region). The spectrophotometric data can reflect different changes in tissue caused by neural activation, including muscle movement or sub-movement (and in some cases pre-movement) changes. These signals can be provided as outputs by the apparatus described herein for use as control signals, including for interfacing with computers and / or devices in both the real world and / or virtual and / or augmented reality.

[0105] The optical properties of tissue that can be detected by the spectrophotometric sensor groups described herein can vary due to one or more factors. Applicants have discovered that these changes are wavelength dependent and can be caused by a small number of sources including heart rate and voluntary and / or involuntary movement (e.g., action, exercise, breathing, speaking, etc.). Changes in absorption, transmission, and / or reflection of light that occur due to muscle movement can cause changes in blood oxygenation, blood flow, and vascular structure (via deformation) due to muscle activation / contraction. The methods and devices described herein can detect these activations and / or contractions as spectrophotometric data.

[0106] Voluntary muscle movements, including those related to locomotion, posture, and balance, can be accurately estimated using spectrophotometric signals from optical sensors strategically clustered on the body (e.g., on or above the skin in communication with activated nerves and / or muscles). Most motor gestures involve a series of muscle and tendon movements that compress arterial geometry to varying degrees, resulting in significant changes in tissue geometry, local blood flow, and blood chemistry (e.g., oxygen saturation). A key advance in this approach is that spatiotemporal variations in tissue absorption near the sensor array can be used to decode motion at distal locations of the body. For example, signals from a device placed on an arm or wrist can be used to decode the motion state of a single finger, even though the "motion" may be many centimeters away from the sensor location. Different kinds of motion result in unique, robust, yet highly repeatable changes in local tissue optical properties (e.g., absorption, transmission, and / or reflection), which result in signals that vary in both intensity and duration. With several nearby sensors, this signal can be used to accurately decode muscle state in real time with incredibly fine precision associated with subtle movements (e.g., finger twitches or eye / gaze position). By subtracting the common components of the signal from the individual channels, large global fluctuations such as the heartbeat can be easily separated.

[0107] During the calibration phase discussed above and in a later interpretation phase in operation, the device can decode spectrophotometric data to detect activation and / or movement of one or more muscle regions. In any of the devices described herein, the device can be trained by comparing intentional movement with real data measurements (such as a camera that tracks body movement), or by directly tracking muscle movement (e.g., by placing fiducial markers / points on the muscle and then using video data to measure the displacement of these points when the muscle is active, or using force sensing resistors, as in traditional myography). In some examples, the training can be used by comparing with traditional neuromuscular sensing signals (e.g., EMG). As described above, the device can be trained using "cue" tasks, for example, instructing a person to move in a specific way, and aligning the neuro-optical signal with the onset of voluntary movement. In some examples, the device can be trained by electrically stimulating specific muscle groups / subgroups using surface electrodes and simultaneously acquiring neuro-optical signals.

[0108] In any of these examples, the signal from the spectrophotometric sensor (including the spectrophotometric representation) can be compared to these other characterization data in the time domain (e.g., using signal decomposition to find independent or principal components in the optical signal that correspond to or correlate with the "real data" signals listed above). This signal decomposition can give a 1:1 relationship between the components of the optical signal and specific muscle activations. Thus, in some examples, machine learning methods and / or regression can be used to find an appropriate matrix or transformation to multiply the optical signal matrix in the time domain to give a muscle activity matrix. This statistical relationship can be determined from a trained model and can be included as (or as part of) the above index. This relationship can be configured as a transformation matrix (index) that can take any incoming signal and transform it into / correlate it to muscle movement. Each transformed muscle movement can then be used as an independent degree of freedom for controlling a device or virtual input (e.g., avatar movement).

[0109] For example, in a simple case, the device may be configured as a headband that can detect muscle movements (or predetermined sequences of muscle movements) of the head or face, such as raising an eyebrow (or a sequence of raising and lowering an eyebrow) to turn on or control the lighting of a headlamp. The methods and devices described herein have been successfully used to determine finger or hand movements with sufficient fidelity to manipulate a cursor on a computer screen and to perform scrolling and clicking actions.

[0110] While the detected positions and / or movements may be specific intentional movements (e.g., driven by intentional movements controlled by the user / wearer), in some examples, these movements may be subconscious, involuntary, or "unintentional" movements. For example, "unintentional" (related to emotions, illness, individual-specific characteristics) muscle activation patterns can be detected and used as input.

[0111] Example Figure 2 and FIG. 3A to FIG. 3C An example of an apparatus (configured as a system) for detecting spectrophotometric data indicative of finger / hand movement is shown. For example, Figure 2 An example of a user's hand and forearm is shown, showing a plurality of spectrophotometric sensor groups 303 (eg, an array) and ... Figure 3A , Figure 3B and / or Figure 3C In some examples, the same sensor (spectrophotometric sensor) can be used to sense and determine a fiducial (e.g., a landmark point) that can be used to adjust the registration of the spectrophotometric representation as described herein. For example, in Figure 2In the embodiment, the reference sensor 311 may also be a spectrophotometric sensor, and the indicated spectrophotometric sensor 303 may also be a reference sensor; in essence, the spectrophotometric representation may include two pieces of information that may be used as a reference between different spectrophotometric representations to align or register subsequent spectrophotometric representations to the calibrated spectrophotometric representation. For example, Figure 2 The image 255 in the figure may include both spectrophotometric sensor data reflecting optical properties that may vary with the position and / or motion of the body part being tracked, and may also include static (or relatively static) parts having optical properties that may be used as fiducial landmarks, such as skin texture 258, blood vessels 256, etc. In these examples, changes in light absorption, light reflection, or light transmission of tissue reflecting subtle autonomous motion may produce a spectrophotometric representation that may be used to determine the position / motion of the body part, and the detected position / motion may be used directly as input for computer control. For example, a mouse cursor may be controlled by: directly from a finger located at a forearm ( Figure 2 ) to form a spectrophotometric representation for moving the X,Y displacement of a computer cursor and for “clicking”. The higher density of sensor readouts can be used to capture high-dimensional control signals associated with arm and finger movements, or for high-fidelity inference of body position and movement in virtual space (Figures 6E1-6E4). The latter can be achieved by using a decoder that establishes a statistical relationship between sensor data (spectrophotometric representations / images) and body positions (e.g., body positions where real data was captured with a camera). Once trained and calibrated, this decoder can be repeatedly realigned across various sessions (e.g., Fig.6D shown) and are used for control purposes.

[0112] The technology can be directly used in consumer devices as an intuitive interface with computers and other electronic devices to convert user-customized natural movements into control signals. These methods and devices can also or alternatively be used to recreate body parts such as hands, arms, legs, eye positions, facial expressions, imagined or active speech in virtual environments (AR, VR, as shown in Figures 6E1-6E4). The statistics of muscle movement are individual-specific characteristics in two anatomical layouts (such as Figure 2 ) can be used as a unique identifier for privacy purposes when in close proximity to a device of interest (e.g., unlocking a phone / computer or door, starting a vehicle, etc.).

[0113] Figure 2 and FIG. 3A to FIG. 3CExamples of devices showing different spectrophotometric sensors and reference sensors are shown. However, as described above, in any of these examples, the same spectrophotometric sensor can be used as the reference sensor. In particular, the same spectrophotometric representation can be used to determine the position / motion of a body region and for providing reference landmarks and for registering the spectrophotometric representation to account for any relative motion of the device (relative to the skin) and / or changes in the skin surface when the device is worn by a user.

[0114] Thus, a non-limiting list of examples of applications in which these methods and apparatus may be used include, for example, decoding of body movements, continuous gesture classification and decoding, eye tracking (face placement), speech decoding (jaw, neck, tongue or face placement), arm / hand / finger movement decoding, leg / foot / toe movement decoding, machine control (devices, IoT, robots), body state decoding, emotional state classification (sleeping vs. awake, excited, happy, nervous, aroused, frustrated), reaction classification (surprised, disappointed, etc.), subthreshold movement / posture / gesture decoding / classification, and intended action decoding (including for use with prostheses), e.g., prosthesis control. As noted above, this "decoding" may be used directly or indirectly to control a device, and / or may be used to recreate a physical or emotional state in virtual reality or augmented reality (VR / AR).

[0115] Additional uses may include, for example, sign language translation ("speech" to sound / text) and medical use cases such as (but not limited to): computer-machine interfaces, early intervention / diagnosis, and / or motor rehabilitation. For example, computer / machine interfaces may include decoding very subtle movements, meaning that patients with mild to severe motor impairments who retain some voluntary movement (e.g., broken arms, carpal tunnel, and in extreme cases of chronic motor impairments due to paraplegia, ALS, and stroke) may use the technology as a customized interface with computers and machines. Early intervention / diagnosis may include most brain and psychiatric disorders with hallmark motor symptoms. An example is Parkinson's disease. Often, treatment / assessment is not sought until these symptoms are severe, and at that stage, modern therapies are not very effective. When used in a consumer setting, detecting the characteristics of muscle movement that are hallmarks of these diseases can serve as a critical early warning for early intervention.

[0116] By using the sensor as the sensory side of a "bio-amplifier", there are also significant advantages when using this technology in a closed loop setting to restore movement or desired physical / emotional states. Electrical stimulation of the muscles can be used with the spectrophotometric sign as the control or optimization signal in the closed loop.

[0117] When compared to existing techniques for detecting the activation of muscles, which typically include electromyography (EMG), the spectrophotometric techniques and apparatus described herein (including the use of optical myography, OMG) have many advantages over traditional EMG techniques for assessing and recording electrical activity generated by skeletal muscle. Both spectrophotometric and EMG methods have the advantage of non-invasive data displays (readouts) of the peripheral nervous system. However, one practical difference is that they operate on different time scales. EMG measures motor unit activity on the order of milliseconds. Spectrophotometric signals can be slower, with their signals evolving at the speed of the muscle, on time scales on the order of 10ms to 100ms. The spectrophotometric signal can reflect muscle activation, which essentially reflects the convolutional activity of many motor units (which produces the EMG signal that drives the muscle activity). Since motor units innervate muscles, and the spectrophotometric signal is ultimately caused by changes in the optical characteristics of the tissue induced by the muscle, these signals may be functionally related. Assuming dense sampling, the spectrophotometric signal can completely reconstruct the EMG signal (e.g., see the more detailed description below). Fig. 6A and Figure 6C ),vice versa.

[0118] Compared with the OMG signal (e.g., spectrophotometric signal) described in this article, the EMG signal has several disadvantages that make it difficult to use outside a highly controlled laboratory / medical environment. First, the EMG signal is highly sensitive to changes in skin impedance (due to sweat, skin elasticity, dust / dirt, dead skin cells, etc.), and this exacerbates signal artifacts. EMG is also sensitive to broadband RF interference (e.g., 60Hz power line noise), which is instantaneous and can be changed by several orders of magnitude in amplitude using a body such as an RF antenna / receiver. The amplification of EMG is complex and expensive, and signal processing is also very complex. Proper signal grounding is also not trivial. In addition, EMG cannot be used at the exact same time as the electrical stimulation of the muscle because the injected current interferes with muscle electrical sensing. Therefore, many applications for simultaneous recording / stimulation of the peripheral nervous system remain extremely challenging. Finally, the decoder that establishes a connection between EMG and body movement is very susceptible to motion artifacts. Even a slight change in the position of the sensor can cause the decoder to become unreliable. In addition, it has not been proven that the EMG activity can be realigned after any subtle or significant movement occurs. Devices using EMG are also less likely to use the same decoder after being repositioned or removed and then reattached. It is worth noting that due to the advantages of OMG, the combined use of EMG and OMG can overcome the problems of motion artifacts and repositioning associated with EMG.

[0119] Spectrophotometric measurements as described herein can be easily performed with simple, readily available off-the-shelf components and integrated circuits (ICs). Spectrophotometric sensors are significantly smaller and cheaper than EMG (this can be on the order of 10x to 1000x in both size and cost). The reduction in cost and extreme miniaturization can capture the relevant market pressures for consumer electronics devices related to the mobile phone industry.

[0120] As described herein, spectrophotometric signals operate at relevant time scales for intuitive control (at the speed of muscle fibers, not motor units). To capture, spectrophotometric signals require lower sampling and potentially lower dynamic range than EMG (appropriate deconvolution can be performed at 10 bits / sample at 100 Hz, compared to 900 Hz and 14 bits / sample for EMG). For example, the inventors have discovered highly accurate decoding using a sensor consisting of dual emitters either red and or IR light and a CMOS sampling 300X200 pixels at 20 Hz and 8-bit accuracy ( Fig. 6E and Fig. 6F ). The amplitude of the spectrophotometric signal is much larger and does not require standard low-noise amplification beyond that of conventional consumer and / or hobby-grade electronics. Furthermore, unlike EMG, which can only sense peripheral electrical activity, the spectrophotometric signal can be used in conjunction with a lens to focus / collect light from deeper muscles, allowing for rapid volume reconstruction of relatively deep muscle tissue.

[0121] Therefore, spectrophotometric signals are ideal for use in most practical human-computer / machine type applications, where EMG might otherwise be used. In the simplest use case, sensors that might be suitable for this purpose have been routinely installed in many consumer devices to detect heart rate (e.g., CMOS-based cameras and green, red, and / or IR light pulse oximeters are commonly used in commercial wearable products), and some modifications may be used according to the description herein. In addition, spectrophotometric signals can be used as a supplement or complete replacement for several medical applications, and even allow the implementation of several medical applications (i.e., brain-computer interfaces or brain-machine interfaces) in which EMG has not yet been proven to be a reliable control signal. Although the most obvious application is in the context of EMG's current inherent signal limitations making it too noisy and / or unreliable, the unique advantages of spectrophotometric signals (e.g., OMG signals) allow these methods and devices to be used.

[0122] FIG. 3A to FIG. 3C An example of an apparatus as described herein is shown. Figure 3A In the embodiment, the device 300 includes a support 309 configured as a strap or cuff that can be secured to a subject's arm, forearm, wrist, etc. Figure 3A, a support 309 includes a plurality of spectrophotometric sensor groups 303 along the inner (skin-facing) side of the strip and possible reference sensors 311, 311'. The device 300 also includes a processor 305, which includes or is coupled to an output 307. The processor and / or the output may be within a housing attached to the support.

[0123] FIG. 3B to FIG. 3C Shown with Figure 3A Compared to a similar view of a strip or hoop being unfolded. Figure 3B In the example, the device 300' also includes a plurality of spectrophotometric sensor groups 303 rigidly attached to a strip 309' (the plurality of spectrophotometric sensor groups 303 can also be configured as reference sensors, as the same data can include reference markers), as well as a processor 305 and an output 307 (e.g., a wireless output). The strip in this example is configured as a watch band structure, but other example structures can be used.

[0124] For example, in any of these examples, several emitters / sensors (e.g., a spectrophotometric sensor group) can generate light of a specific wavelength and can measure optical signals from nearby tissue. Several sensors / emitters can be connected to a single microprocessor. The microprocessor can be connected to a computer via a wired tether (e.g., USB, Ethernet) or wirelessly (e.g., Wi-Fi, Bluetooth, RF). Software on the computer can store incoming data, and the data can also be used as control signals for interacting with the computer or with a device that is also connected to the computer (using a wired or wireless connection). Alternatively, the microprocessor can be used to directly control the device. The processor can be bidirectionally connected to a remote server, streaming data for further processing, storage, or interaction with a remote device.

[0125] Any suitable sensor layout can be used. In one example, sensors (spectrophotometric sensor groups) can be distributed in close proximity on the forearm (e.g., about 15mm to 20mm apart) to detect changes in the optical properties of tissue caused by distal movements of the wrist and fingers. Light can be emitted into the tissue by the spectrophotometric sensor group, and some of this light will be received by nearby sensors and converted into changes in voltage, depending on the optical properties of the tissue. The analog voltage signal can then be amplified and digitized. Next, the data can be preprocessed by applying temporal low-pass, high-pass, and anti-aliasing filters to the incoming data. Specific spatiotemporal features of time series data can be used to identify and separate noise and signal artifacts. Several sensors are connected to a single microprocessor, and the average of several sensors is calculated and subtracted from each channel. The signal can be used directly as a control signal, or alternatively, the computer software will further process the data by applying a decoder or classifier to relate the incoming signal to a specific movement or intended action, and this can then be used to control a device or the computer itself that is also connected to the computer (using a wired connection or a wireless connection). Thus, typically, the sensor data can be used directly as a control signal and / or can be further processed for use as a control signal. The sensor data and decoded / classified output can also be streamed bidirectionally to a remote server for further processing, storage, or interaction with an Internet-connected device.

[0126] Figure 4 An example of a process for processing spectrophotometric sensor data is shown. In this example, the raw (unprocessed) spectrophotometric sensor output 404 can be processed during a preprocessing stage, for example, for amplification, filtering, signal subtraction (e.g., of a heartbeat), digitization, etc. 404. The preprocessing can be integrated with the spectrophotometric sensor group or can be separate. In some examples, the preprocessing can be integrated with the processor, or the preprocessing can be coupled to the processor but distinct from the processor. In some examples, the preprocessing can include registering the spectrophotometric sensor (and / or registering the spectrophotometric representation obtained by the spectrophotometric sensor) 406. Processing that can be performed locally and / or remotely can include using the above-described indexes to match spectrophotometric data with muscle movements, including identifying the position and / or movement of a body part (in some examples, a muscle of a body part) as being associated with a particular sensed spectrophotometric representation, and / or identifying how activation patterns from the spectrophotometric representation should be interpreted as control signals 408 to control a device, computer, and / or software that receives an output indicator of the position and / or movement of a body part (or an indicator of that movement).

[0127] Figure 5 and FIG. 6A to FIG. 6C Spectrophotometric data obtained from the example device described above is shown. Figure 5In the example shown, OMG data is from a set of 4 spectrophotometric sensors placed radially around the left forearm near the wrist in close proximity (approximately 1 cm to 2 cm) (see, for example, Figure 2 and Figure 3A ). The array of these four sensors can be a spectrophotometric representation at a given time. Figure 5 In Figure 5, each line (505, 507, 509, 511) reflects data from a different sensor. The left ring finger was tapped 10 times at approximately 1 Hz with slightly different intensities, and the process was then repeated three times with the index finger. The top bars represent the taps of the ring and index fingers, respectively. Each sensor has a 0.1 Hz high pass filter, and common mode rejection is used to separate the signal corresponding to the heartbeat.

[0128] Fig. 6A Shown from (such as Figure 5 The data from 4 nearby sensors in the figure are synchronized electromyography (EMG) records. The left ring finger taps 5 times at a frequency of about 1Hz. Figure 5 As shown, the bars represent the duration of the finger tapping. Figure 6B An overlay of 4 channels of sensor data for tapping the index finger (n=10 taps) or ring finger (n=5 taps) is shown, showing remarkable repeatability of the signal for the same motion. Thus, the spectrophotometric signal can be used to determine specific finger motion and / or position.

[0129] Figure 6C A comparison between OMG and EMG from a single channel is shown, both positioned on the middle of the forearm and aligned with about 200 repetitions of an exaggerated grasping motion. The top two graphs depict heat maps, while the bottom graph shows each trial overlay, both aligned with the grasp. The data show an example of a spectrophotometric representation (in this case, a spectrophotometric movie) that can be used to determine position and / or motion (e.g., tapping) with high accuracy and precision as described herein.

[0130] Fig.6D Multiple frames showing real-time reconstruction of hand and finger keypoint locations (right) compared to ground truth using OMG data (left).

[0131] Figures 6E1 to 6E4 illustrate performance metrics for continuous decoding. The top two panels (Figures 6A1 to 6E2) show predicted performance real-time decoding of a single keypoint, specifically the tip of the index finger, for two separate decoding sessions. In both panels, the darker line 618 represents the true data recorded by the camera, while the lighter line 620 represents the estimated keypoint location. The lower left panel (Figure 6E3) shows OMG data obtained from the forearm aligned with a force sensor that is synchronized with a prompt prompting the user to press the sensor with approximately equal force. The lighter line represents the change in resistance from the force sensor, while the darker line represents the OMG from a single sensor. The shading in this panel represents the standard deviation of the mean of 100 trials. The lower right panel (Figure 6E4) depicts real-time decoding accuracy metrics for retained data of all 3D keypoints of the right hand after the model was trained based on OMG data taken from the right wrist. The units are Pearson correlation coefficient R, and the shading represents the X, Y, or Z domain.

[0132] As mentioned, any of these methods and devices can adjust the registration during operation of the method or device. In particular, the methods and devices described herein can use the spectrophotometric representations recorded during operation to adjust the registration of the device and determine the position and / or movement of the body region. Fig. 6F An example of a session-to-session alignment (registration) technique that can be used is provided. In a first session, data obtained from a spectrophotometric sensor (spectrophotometric representations) is decomposed into optical property signals, and these signals can be used together with real data images (e.g., images of body part positions and / or motions) to train a model (e.g., a machine learning agent) that relates OMG to hand or finger positions. Next, in a second session in which the device can be repositioned or shifted relative to the patient's skin, data can be collected again (e.g., spectrophotometric representations are collected). This data can then be aligned with the first session using an alignment technique, such as an image registration technique that uses multiple spectrophotometric representations to transform the spectrophotometric representations so that they are registered with previously acquired spectrophotometric representations. The registered data can also be decomposed into optical property signals (spectrophotometric representations), and the previous model can now be implemented to accurately estimate hand and / or finger positions.

[0133] The use of registration, particularly with respect to spectrophotometric representations, can be particularly useful when determining the position and / or movement of body parts (e.g., hands, fingers, etc.). Figure 6G is shown in Figure 6GA comparison of average correlation data without registration (showing very low correlation) (middle graph) and average correlation data with registration (showing very high correlation) (rightmost graph) is shown. The graph on the left shows the initial (calibrated) data. In some cases, using registration to adjust the registration of the spectrophotometric representation can result in greatly improved output.

[0134] Figure 7 A method using any of the devices described herein is illustrated. In the figure, a device (e.g., which may include one or more high-density spectrophotometric sensors) is initially positioned on a skin region near (but separated from) a body region to be monitored, and an optical property signal 701 can be acquired. Next, a model is constructed to correlate the optical property signal with real data (as described above, using cameras and posture estimation to gather key points about hand joint positions) 703. In some examples, a fiducial marker (marker point) can be identified from the OMG data (test or training spectrophotometric representation), which can be derived from the skin (e.g., wrinkles, freckles, moles, blemishes, tendons, blood vessels, etc.). The marker point can be within the spectrophotometric representation and can be anything that causes contrast or changes in a static (i.e., a single time point) optical property signal (spectrophotometric representation). These markers that change over time as the user moves can then be used to create a spatiotemporal template. The template can be used to align data within and across sessions. Furthermore, in situations where the sensor may be repositioned due to user motion, opening and closing the device again, etc., the template can then be used to align the optical property signal data (spectrophotometric representation) from subsequent sessions.

[0135] Figure 8 Examples of methods of using any of these devices as described herein are shown. Figure 8 In the present invention, a device (e.g., a spectrophotometric sensor) may initially be placed on or near a body part to be monitored (e.g., or a tendon or muscle coupled to a body part to be monitored) 801. A spectrophotometric (i.e., OMG) sensor group may then be used to detect optical characteristic signals from the spectrophotometric sensor group 803. As part of or prior to this process, a calibration step may be performed as described above, and each spectrophotometric sensor group may be associated with a particular muscle and / or movement (or multiple muscles and / or multiple movements).

[0136] The spectrophotometric signal may be pre-processed or processed (e.g., to separate out the heartbeat or any other global optical signal) 805. In general, the device may process the spectrophotometric signal to determine which muscle or movement is associated with the spectrophotometric signal 807. The result may be output as an indicator of muscle movement 809 based on the processed signal.

[0137] It should be appreciated that all combinations of the foregoing concepts and additional concepts discussed in greater detail below (provided such concepts are not mutually inconsistent) are considered to be part of the inventive subject matter disclosed herein and can be used to achieve the benefits described herein.

[0138] The process parameters and step sequences described and / or illustrated herein are provided as examples only and may be changed as desired. For example, although the steps shown and / or described herein may be shown or discussed in a particular order, the steps do not necessarily need to be performed in the order shown or discussed. The various example methods described and / or illustrated herein may also omit one or more steps described or illustrated herein, or include additional steps in addition to the disclosed steps.

[0139] Fig. 9 Another example of a method as described herein is schematically shown. Fig. 9 In some embodiments, the method may be used to determine the position and / or movement of a region of a subject's body (e.g., a finger, hand, etc.) and / or the force exerted by the region by first obtaining a spectrophotometric representation of an area of ​​the subject's skin (e.g., by collecting data from each photometric sensor in an array of spectrophotometric sensors) 901. In any of these examples, the method may be to construct or adapt a model 900, as described above. The method may then include preprocessing as described above, which may include separating and / or amplifying, filtering, etc. the photometric sensor data from the heartbeat signal data 903. The preprocessing may include adjusting the alignment of the spectrophotometric representation by comparing the spectrophotometric representation of the region of the subject's skin to a previous spectrophotometric representation of the region of the subject's skin 905, and transforming the spectrophotometric representation using an appropriate technique as described above to form an adjusted spectrophotometric representation of the region of the subject's skin. The method may then use the spectrophotometric representation (e.g., adjust the spectrophotometric representation) to determine the position and / or motion of a region of the subject's body and / or a force exerted by the region 907, and output the identified position and / or motion 909. As described above, the output may be used to control a device, such as input to a computer, etc.

[0140] In some examples, determining or decoding the position of a body region (e.g., hand position, finger position, etc.) can be performed as described herein, and pose estimation can be performed based on the optical characteristic signals (spectrophotometric representation). For example, an initial phase can include collecting high-quality data from both an OMG sensor and a synchronized image (e.g., a video feed). For example, both OMG (spectrophotometric) data and images of the body region to be tracked can be captured simultaneously while the subject performs various movements (in some examples, finger and hand movements). A system including a spectrophotometric sensor can record optical characteristic signals corresponding to the observed movements, while the video feed visually captures the corresponding hand and finger movements.

[0141] The spectrophotometric data (OMG data) and the corresponding image can be synchronized (in time), ensuring that the optical property signal is associated with the correct frame from the video feed. Optionally, the position or motion can be marked (with an identifier of the position or motion). Data preprocessing can then be performed on both data modalities. For image (e.g., video) data, this can include cropping, resizing, and normalizing the image to focus on the hand and fingers. For OMG data (e.g., spectrophotometric data), filtering techniques can be applied to remove noise and separate other non-essential components (such as heartbeat-related signals) from the signal.

[0142] Next, in some examples, computer vision techniques can be used to extract the coordinates of key points (e.g., finger joints and fingertips) from each video frame. These key points can be used to form "real data" labels for corresponding OMG signals for training machine learning agents, or for forming or refining analytical models.

[0143] For example, the acquired data can be used to train a machine learning agent. For example, a machine learning model can be trained to learn the mapping between OMG signals (spectrophotometric representations) and ground truth data labels obtained from pose estimation. In some cases, this may be a regression problem where the model aims to predict continuous output variables (keypoints) from the input OMG data. Methods such as gradient boosting regression, convolutional neural networks (CNNs), and / or recurrent neural networks (RNNs) can be utilized as they have been shown to be effective in learning on time series and image data.

[0144] After the training process, the performance of the model can be evaluated using a different test set that includes data that was not used during the training phase. Metrics such as the mean squared error or mean absolute error between the model's predictions and the true data labels can be used to quantify the accuracy of the model.

[0145] Finally, the trained model can be deployed. In a live environment, the model can receive real-time OMG data, process it, and generate predictions of hand and finger positions accordingly. These predictions can then be used for a variety of applications, including but not limited to gesture recognition, prosthetic control, and human-computer interaction.

[0146] The methods and apparatus described herein may refer to this as pose estimation using the above techniques. Pose estimation, in which the position and / or motion of a body region is evaluated on a continuous basis, has many advantages over other techniques that may alternatively use gesture recognition. The techniques and apparatus described herein use optical property signals (e.g., spectrophotometric representations) to continuously decode positions ("poses"), such as hand poses, which can provide real-time positions of all joints in a body region (e.g., fingers, hands, etc.). In general, pose estimation (e.g., hand pose estimation) is a challenging problem related to but different from gesture recognition. Gesture recognition involves recognizing specific predefined hand movements or configurations, such as waving, giving a thumbs up, or a sign in sign language. This can typically involve training a machine learning model based on a set of labeled examples for each gesture, and then using the model to classify new examples into one of the known categories.

[0147] On the other hand, hand pose estimation is a more general and complex problem. It involves identifying the position and orientation of hands and fingers in an image, regardless of what gestures they may form. This may require identifying the hands and their key points (such as fingertips and joints) and estimating their 3D coordinates. This can then be used, for example, to track the movement of hands and fingers over time in real time. The complexity of these problems depends on various factors, such as the diversity and ambiguity of gestures, the quality and diversity of training data, the complexity of the background in which the hands are imaged, and the speed and accuracy requirements of the application. However, in general, hand pose estimation is considered a more difficult problem than gesture recognition. This is partly due to the higher dimensionality of hand pose estimation. Pose estimation deals with a large number of variables. For example, if each hand has 21 key points and each key point has 3 coordinates (x, y and depth), then a total of 63 variables need to be estimated for each image. In contrast, gesture recognition only requires classifying the image into one of several categories. In addition, pose estimation as described in this article can provide continuous outputs. In hand pose estimation, the output is a set of continuous variables (e.g., the coordinates of key points), which makes the problem more complicated. On the other hand, gesture recognition is a classification problem with discrete outputs.

[0148] Finally, the pose estimation techniques described in this article provide real-time performance, giving accurate, real-time hand pose estimates with high precision and speed, which is particularly helpful for applications such as control-based AR / VR, gaming, or sign language interpretation. This requires more complex and computationally demanding models than gesture recognition. The complexity of hand pose estimation is several orders of magnitude greater than gesture recognition, but provides significant advantages as described in this article.

[0149] Force determination In addition to determining the position and / or movement of a body part, the methods and apparatus described herein can also calculate, determine, or estimate the forces exerted by the body part relative to the environment. One or more spectrophotometric representations as described herein can be collected using optical myography as described herein; the spectrophotometric representations include information representing the forces exerted by muscles and tendons to drive the movement of a body part (e.g., a finger, arm, hand, etc.). This information is not available in other modalities (such as imaging data).

[0150] Forces applied by a body region may be identified similarly to position and / or motion, and any of the methods and apparatus described herein may include interpreting a spectrophotometric representation using a model, such as a statistical model and / or a machine learning model (e.g., a neural network) to determine the forces applied by the body region. For example, a machine learning agent may be trained using a test spectrophotometric representation that includes corresponding images showing strain gauge readings when a body region applies a force; for example, a training data set may include one or more images showing the motion of a test subject (or subjects) applying a force to an object and an indicator of the applied force (from a strain gauge or other force sensor). Force estimates may be provided directly from a force gauge or strain gauge and / or may be determined indirectly, for example, by measuring one or more indicators of the applied force (e.g., deflection or deformation of a material, etc.). After training, a model (e.g., a machine learning model) can be used as part of a method or apparatus to interpret the current activity of a body part (e.g., a finger, hand, wrist, arm, etc.) and output forces applied to the body part in addition to or in lieu of position and / or motion.

[0151] Generally, in any of the methods and devices described herein, where the position and / or movement of a reference body region or body part is described, it should be understood that the system and / or method may also or alternatively be configured to determine and / or output forces exerted by the body part.

[0152] Any of the methods described herein (including user interfaces) may be implemented as software, hardware, or firmware, and may be described as a non-transitory computer-readable storage medium storing a set of instructions executable by a processor (e.g., a computer, tablet, smart phone, etc.), which, when executed by the processor, causes the processor to control the execution of any steps, including but not limited to: display, communication with a user, analysis, modification of parameters (including timing, frequency, intensity, etc.), determination, reminders, etc. For example, any of the methods described herein may be performed at least in part by a device including one or more processors having a memory storing a non-transitory computer-readable storage medium storing a set of instructions for the process of the method.

[0153] Although various embodiments have been described and / or illustrated herein in the context of a fully functional computing system, one or more of these exemplary embodiments may be distributed in various forms as a program product, regardless of the specific type of computer-readable medium used to actually perform the distribution. The embodiments disclosed herein may also be implemented using software modules that perform certain tasks. These software modules may include scripts, batch processing, or other executable files that may be stored in a computer-readable storage medium or a computing system. In some embodiments, these software modules may configure a computing system to perform one or more example embodiments disclosed herein.

[0154] As described herein, the computing devices and systems described and / or illustrated herein broadly represent any type or form of computing device or system capable of executing computer-readable instructions (such as those contained within the modules described herein). In their most basic configuration, these computing devices can each include at least one memory device and at least one physical processor.

[0155] The term "memory" or "memory device" as used herein generally refers to any type or form of volatile or non-volatile storage device or medium capable of storing data and / or computer-readable instructions. In one example, a memory device can store, load and / or maintain one or more modules described herein. Examples of memory devices include, but are not limited to, random access memory (RAM), read-only memory (ROM), flash memory, hard disk drive (HDD), solid-state drive (SSD), optical drive, cache, a variation or combination of one or more of them, or any other suitable storage memory (storage memory).

[0156] In addition, the term "processor" or "physical processor" as used herein generally refers to any type or form of hardware-implemented processing unit capable of interpreting and / or executing computer-readable instructions. In one example, the physical processor can access and / or modify one or more modules stored in the above-mentioned memory device. Examples of physical processors include, but are not limited to, microprocessors, microcontrollers, central processing units (CPUs), field programmable gate arrays (FPGAs) implementing soft-core processors, application-specific integrated circuits (ASICs), parts of one or more of them, variations or combinations of one or more of them, or any other suitable physical processor.

[0157] Although shown as separate elements, the method steps described and / or illustrated herein may represent portions of a single application. Additionally, in some embodiments, one or more of these steps may represent or correspond to one or more software applications or programs that, when executed by a computing device, may cause the computing device to perform one or more tasks, such as method steps.

[0158] Furthermore, one or more of the devices described herein may convert data, physical devices, and / or representations of physical devices from one form to another. Additionally or alternatively, one or more modules described herein may convert a processor, volatile memory, non-volatile memory, and / or any other portion of a physical computing device from one form of computing device to another form of computing device by executing on the computing device, storing data on the computing device, and / or otherwise interacting with the computing device.

[0159] The term "computer-readable medium" as used herein generally refers to any form of device, carrier or medium capable of storing or carrying computer-readable instructions. Examples of computer-readable media include, but are not limited to, transmission-type media (such as carrier waves) and non-transitory types of media, such as magnetic storage media (e.g., hard drives, tape drives, and floppy disks), optical storage media (e.g., compact disks (CDs), digital video disks (DVDs), and BLU-RAY disks), electronic storage media (e.g., solid-state drives and flash memory media), and other distribution systems.

[0160] Those of ordinary skill in the art will recognize that any process or method disclosed herein may be modified in a variety of ways. The process parameters and order of steps described and / or illustrated herein are given as examples only and may be changed as needed. For example, although the steps illustrated and / or described herein may be shown or discussed in a particular order, the steps do not necessarily need to be performed in the order illustrated or discussed.

[0161] The various exemplary methods described and / or shown herein may also omit one or more steps described or shown herein, or include additional steps in addition to these disclosed steps. In addition, the steps of any method as disclosed herein may be combined with any one or more steps of any other method as disclosed herein.

[0162] A processor as described herein may be configured to perform one or more steps of any method disclosed herein.Alternatively or in combination, a processor may be configured to combine one or more steps of one or more methods as disclosed herein.

[0163] When a feature or element is referred to herein as "on another feature or element", it can be directly on another feature or element, or there can also be intermediate features and / or elements. In contrast, when a feature or element is referred to as "directly on another feature or element", there are no intermediate features or elements. It will also be understood that when a feature or element is referred to as "connected", "attached" or "coupled" to another feature or element, it can be directly connected, attached or coupled to the other feature or element, or there can be intermediate features or elements. In contrast, when a feature or element is referred to as "directly connected", "directly attached" or "directly coupled" to another feature or element, there are no intermediate features or elements. Although described or shown with respect to one embodiment, the features and elements described or shown in this way can be applied to other embodiments. Those skilled in the art will also recognize that the reference to a structure or feature that is set to be "adjacent" to another feature can have a portion that covers adjacent features or is located below adjacent features.

[0164] The terms used herein are only used for the purpose of describing specific embodiments and are not intended to limit the present invention. For example, as used herein, the singular forms "a", "an" and "the" are also intended to include the plural forms, unless the context clearly indicates otherwise. It will be further understood that when used in this specification, the terms "comprises" and / or "comprising" specify the presence of stated features, steps, operations, elements and / or parts, but do not exclude the presence or addition of one or more other features, steps, operations, elements, parts and / or groups thereof. As used herein, the term "and / or" includes any and all combinations of one or more of the associated listed items and may be abbreviated as " / ".

[0165] Spatially relative terms such as "under", "below", "lower", "over", "upper" and similar terms may be used herein to easily describe the relationship of one element or feature to another (or more) elements or features as shown in the drawings. It will be understood that spatially relative terms are intended to include different orientations of the device in use or operation other than the orientation depicted in the drawings. For example, if the device in the drawings is inverted, the elements described as "below" or "below" other elements or features will be oriented to be "above" other elements or features. Therefore, the exemplary term "below" can cover both the above and below orientations. The device can be oriented in other ways (rotated 90 degrees or in other orientations), and the spatially relative descriptors used herein are interpreted accordingly. Similarly, the terms "upwardly", "downwardly", "vertical", "horizontal" and similar terms are used herein for explanation purposes only, unless otherwise specifically indicated.

[0166] Although the terms "first" and "second" may be used herein to describe various features / elements (including steps), these features / elements should not be limited by these terms unless the context indicates otherwise. These terms may be used to distinguish one feature / element from another feature / element. Thus, without departing from the teachings of the present invention, the first feature / element discussed below may be referred to as the second feature / element, and similarly, the second feature / element discussed below may be referred to as the first feature / element.

[0167] Throughout this specification and the appended claims, unless the context requires otherwise, the word "comprise" and variations such as "comprises" and "comprising" mean that various components can be used together in methods and articles (e.g., compositions and devices including apparatuses and methods). For example, the term "comprising" will be understood to mean the inclusion of any stated elements or steps but not the exclusion of any other elements or steps.

[0168] Generally, any apparatus and method described herein should be understood to be inclusive, but all or a subset of components and / or steps may optionally be exclusive and may be expressed as "consisting of various components, steps, sub-components or sub-steps" or optionally "consisting essentially of various components, steps, sub-components or sub-steps."

[0169] As used herein in the specification and claims, including those used in the examples, and unless otherwise expressly specified, all numbers may be considered as if preceded by the words "about" or "approximately", even if the term does not appear explicitly. The phrase "about" or "approximately" may be used when describing amplitude and / or position to indicate that the value and / or position described are within the reasonable expected range of the value and / or position. For example, a numerical value may have a value as + / -0.1% of a stated value (or range of values), + / -1% of a stated value (or range of values), + / -2% of a stated value (or range of values), + / -5% of a stated value (or range of values), + / -10% of a stated value (or range of values), etc. Unless otherwise indicated by the context, any numerical value given herein should also be understood to include about or approximately that value. For example, if the value "10" is disclosed, then "about 10" is also disclosed. Any numerical range listed herein is intended to include all subranges contained therein. It should also be understood that when a value is disclosed, "less than or equal to" the value, "greater than or equal to the value," and possible ranges between the values ​​are also disclosed, as appropriately understood by those skilled in the art. For example, if a value "X" is disclosed, "less than or equal to X" and "greater than or equal to X" (e.g., where X is a numerical value) are also disclosed. It should also be understood that throughout the application, data is provided in a variety of different formats, and that the data represents endpoints and starting points and ranges for any combination of data points. For example, if a specific data point "10" and a specific data point "15" are disclosed, it should be understood that greater than, greater than or equal to, less than, less than or equal to, and equal to 10 and 15, and 10 to 15 are disclosed. It should also be understood that every unit between two specific units is also disclosed. For example, if 10 and 15 are disclosed, 11, 12, 13, and 14 are also disclosed.

[0170] Although various illustrative embodiments are described above, any of several changes may be made to the various embodiments without departing from the scope of the invention as described in the claims. For example, in optional embodiments, the order in which the various described method steps are performed may generally be changed, and in other optional embodiments, one or more method steps may be skipped together. Optional features of various device and system embodiments may be included in some embodiments and not included in other embodiments. Therefore, the foregoing description is provided primarily for exemplary purposes and should not be construed as limiting the scope of the invention as set forth in the claims.

[0171] The examples and descriptions included herein show specific embodiments in which the subject matter can be practiced by way of illustration and not limitation. As mentioned, other embodiments can be utilized and derived therefrom so that structural and logical replacements and changes can be made without departing from the scope of the present disclosure. If in fact more than one is disclosed, for convenience only, such embodiments of the subject matter of the invention can be referred to herein individually or collectively by the term "invention", and it is not intended that the scope of the present application be voluntarily limited to any single invention or inventive concept. Therefore, although specific embodiments have been illustrated and described herein, any arrangement calculated to achieve the same purpose can replace the specific embodiments shown. The present disclosure is intended to cover any and all modifications or variations of various embodiments. After reading the above description, the combination of the above embodiments and other embodiments not specifically described herein will be obvious to those skilled in the art.

Claims

1. A method for determining the position and / or movement of a region of a subject's body by: obtaining a spectrophotometric representation of a region of the subject's skin by collecting data from each photometric sensor in an array of spectrophotometric sensors; adjusting a registration of the spectrophotometric representation of the portion of the subject's skin by comparing the spectrophotometric representation of the portion of the subject's skin to a previous spectrophotometric representation of the portion of the subject's skin to form an adjusted spectrophotometric representation of the portion of the subject's skin; The position and / or motion of a region of the subject's body is determined using the adjusted spectrophotometric representation of the region of the subject's skin.

2. The method of claim 1, further comprising outputting the position and / or movement of a region of the subject's body. 3 . The method of claim 1 , further comprising operating one or more devices based on the determined position and / or movement of the region of the subject's body.

4. The method of claim 1, further comprising operating one or more of a control computer and software based on the determined position and / or movement of the region of the subject's body.

5. The method of claim 1 further comprising repeating the step of obtaining the spectrophotometric representation to determine movement of a region of the subject's body.

6. The method of claim 5, further comprising the step of repeatedly adjusting the representation registration of the spectrophotometric representation each time the spectrophotometric representation is acquired.

7. The method of claim 5, further comprising repeating the step of acquiring the spectrophotometric representation at a frequency of 5 Hz or higher.

8. The method according to claim 1, wherein: Determining the position and / or movement of the region of the subject's body comprises continuously determining the position of the region of the subject's body by repeating the steps of obtaining the spectrophotometric representation and adjusting the registration of the representation.

9. The method according to claim 1, wherein: Obtaining the spectrophotometric representation of a region of the subject's skin includes obtaining a region of the subject's skin covering 1 cm 2 or the spectrophotometric representation of a portion of a larger area.

10. The method according to claim 1, wherein: Adjusting the registration includes using a rigid or non-rigid transformation technique to account for movement of the array of spectrophotometric sensors relative to the subject's skin and / or to account for changes in the shape of the subject's skin.

11. The method according to claim 10, wherein: Adjusting the representation registration includes using a non-rigid transformation technique.

12. The method according to claim 10, wherein: Adjusting the representation registration includes using an affine transformation technique.

13. The method according to claim 1, wherein: The region of the subject's body includes one or more of the subject's hands, fingers, arms, legs, feet, or head.

14. The method according to claim 1, wherein: Determining the position and / or movement of the region of the subject's body includes using a machine learning agent that has been trained using a training data set, the training data set comprising a plurality of previous spectrophotometric representations of the skin and a plurality of video representations, the plurality of previous spectrophotometric representations being acquired using a test array of spectrophotometric sensors, the plurality of video representations showing the region of the test subject's body at a time corresponding to each previous spectrophotometric representation of the plurality of previous spectrophotometric representations.

15. The method according to claim 1, wherein: Determining the position and / or movement of the region of the subject's body includes using a machine learning agent that has been trained using a training data set, the training data set comprising a plurality of previous spectrophotometric representations of the subject's skin and a plurality of video representations, the plurality of previous spectrophotometric representations being acquired using an array of the spectrophotometric sensors, the plurality of video representations showing the region of the subject's body at a time corresponding to each previous spectrophotometric representation of the plurality of previous spectrophotometric representations.

16. The method of claim 15, further comprising training the machine learning agent based on the training data set, the training data set comprising a plurality of spectrophotometric representations of the subject's skin obtained from the array of spectrophotometric sensors and corresponding video representations of regions of the subject's body.

17. The method according to claim 1, wherein: Determining the position and / or movement of a region of the subject's body comprises using a statistical model.

18. The method according to claim 1, wherein: Obtaining the spectrophotometric representation includes obtaining the spectrophotometric representation using a wearable device that holds the array of spectrophotometric sensors against a region of the skin.

19. A method for determining the position and / or movement of a region of a subject's body by: By collecting data from each photometric sensor in an array of spectrophotometric sensors, a 1 cm coverage of the subject's skin is obtained. 2 or a spectrophotometric representation of a larger area; adjusting the registration of the spectrophotometric representation by comparing the spectrophotometric representation to a previous spectrophotometric representation of a region of the subject's skin to account for one or more of: movement of the array of spectrophotometric sensors relative to the subject's skin or changes in the shape of the subject's skin; and The position and / or motion of the region of the subject's body is determined and output using the adjusted spectrophotometric representation of the region of the subject's skin.

20. A system for determining the position and / or movement of a region of a subject's body, the system comprising: - a spectrophotometric sensor set comprising an array of spectrophotometric sensors configured to obtain a spectrophotometric representation of a portion of a subject's skin; - a support configured to hold the set of spectrophotometric sensors adjacent a surface region of the subject's skin; - a control circuit comprising one or more processors; as well as a memory coupled to the one or more processors, the memory storing computer program instructions that, when executed by the one or more processors, are configured to iteratively: -- obtaining the spectrophotometric representation of the region of the subject's skin by collecting data from each spectrophotometric sensor in an array of spectrophotometric sensors; -- adjusting the registration of the spectrophotometric representation by comparing the spectrophotometric representation of the portion of the subject's skin to a previous spectrophotometric representation of the portion of the subject's skin to form an adjusted spectrophotometric representation of the portion of the subject's skin; --Determining and outputting the position and / or motion of the region of the subject's body using the adjusted spectrophotometric representation of the region of the subject's skin.

21. The system of claim 20, wherein: The program instructions, when executed by the one or more processors, are configured to iteratively retrieve the spectrophotometric representation at a frequency of 5 Hz or higher and adjust representation registration.

22. The system of claim 20, wherein: The program instructions, when executed by the one or more processors, are configured to separate a component of the optical characteristic signal corresponding to a heartbeat.

23. The system of claim 20, wherein: The spectrophotometric sensor group includes a light emitter and an optical detector.

24. The system of claim 23, wherein: The light emitter includes one or more of a photodiode and an LED.

25. The system of claim 23, wherein: The optical detector comprises a photodetector.

26. The system of claim 20, further comprising a signal conditioner comprising one or more of a lens, a diffuser, a filter, and a lens array, the signal conditioner configured to modify the spectrophotometric representation of a region of the subject's skin.

27. The system of claim 20, wherein: The support member includes a garment.

28. The system of claim 20, wherein: The support member may comprise a strap, a cuff, a patch or a belt.

29. The system of claim 20, wherein: The support is configured to fit on one or more of a user's forearm, wrist, or hand.

30. The system of claim 20, wherein: The processor is configured to wirelessly output the position and / or movement of the portion of the subject's body using the adjusted spectrophotometric representation of the portion of the subject's skin.

31. The system of claim 20, wherein: The processor comprises a remote processor.

32. The system of claim 20, wherein: The spectrophotometer of the area represents 1 cm of the subject's skin. 2 or larger area.

33. The system of claim 20, wherein: The processor is configured to adjust the representation registration using a rigid or non-rigid transformation technique to account for movement of the array of spectrophotometric sensors relative to the subject's skin and / or to account for changes in the shape of the subject's skin.

34. The system of claim 33, wherein: The processor is configured to adjust the representation registration using a non-rigid transformation.

35. The system of claim 33, wherein: The processor is configured to adjust the representation using an affine transformation.

36. The system of claim 20, wherein: The processor is configured to determine and output the position and / or movement of the region of the subject's body using a machine learning agent, the machine learning agent having been trained using a training data set, the training data set comprising a plurality of previous spectrophotometric representations of the skin and a plurality of video representations, the plurality of previous spectrophotometric representations being acquired using a test array of spectrophotometric sensors, the plurality of video representations showing the region of the test subject's body at a time corresponding to each previous spectrophotometric representation of the plurality of previous spectrophotometric representations.

37. A system for detecting neuromuscular activity, the system comprising: a spectrophotometric sensor set comprising an array of spectrophotometric sensors, wherein each spectrophotometric sensor is configured to detect from tissue a spectrophotometrically representative optical property signal corresponding to a region of the subject's skin; a support configured to hold the spectrophotometric sensor adjacent to a skin surface; and A processor is configured to receive the spectrophotometric representation from the spectrophotometric sensor group and detect muscle movement from the spectrophotometric representation after separating a component of the optical characteristic signal corresponding to the heartbeat from the received optical characteristic signal.

38. The system of claim 37, further wherein: The processor is configured to adjust the registration of the spectrophotometric representation of the portion of the subject's skin by comparing the spectrophotometric representation of the portion of the subject's skin to a previous spectrophotometric representation of the portion of the subject's skin to form an adjusted spectrophotometric representation of the portion of the subject's skin.

39. The system of claim 37, wherein: The processor is configured to separate the component of the optical characteristic signal corresponding to the heartbeat to detect muscle movement.

40. The system of claim 37, wherein: The spectrophotometric sensor group includes a light emitter and an optical detector.

41. The system of claim 37, wherein: The light emitter includes one or more of a photodiode and an LED.

42. The system of claim 37, wherein: The optical detector comprises a photodetector.

43. The system of claim 37, further comprising a plurality of spectrophotometric sensor groups secured by the support.

44. The system of claim 37, further comprising a signal conditioner comprising one or more of a lens, a diffuser, a filter, and a lens array.

45. The system of claim 37, wherein: The support member includes a garment.

46. ​​The system of claim 37, wherein: The support member may comprise a strap, a cuff, a patch or a belt.

47. The system of claim 37, wherein: The support is configured to fit on one or more of a user's forearm, wrist, or hand.

48. The system of claim 37, wherein: The processor is configured to output an indicator of muscle movement.

49. The system of claim 48, wherein: The processor is configured to wirelessly output an indicator of the muscle movement.

50. The system of claim 37, wherein: The processor is configured to associate the spectrophotometric sensor with one or more of muscle movement or body part movement.

51. The system of claim 37, wherein: The processor includes processing circuitry.

52. The system of claim 37, wherein: The processor comprises a remote processor.

53. A wearable system for detecting neuromuscular activity, the system comprising: a plurality of spectrophotometric sensors configured to sense an optical property, wherein the plurality of spectrophotometric sensors include at least one light emitter and a plurality of optical detectors, wherein each spectrophotometric sensor is configured to detect from the tissue an optical property signal that is spectrophotometrically representative of a region of the subject's skin; a support configured to hold the plurality of optical detectors in the spectrophotometric sensor adjacent to a skin surface such that the plurality of optical detectors are arranged in a pattern relative to the skin surface; and A processor is configured to receive an optical characteristic signal from each of the plurality of optical detectors, separate a signal corresponding to a heartbeat from the optical characteristic signal, and distinguish muscle movements corresponding to one or more muscles based on the received optical characteristic signal.

54. The wearable system of claim 53, wherein: The processor is configured to distinguish muscle movement from the received optical characteristic signal.

55. The wearable system of claim 53, wherein: The processor is configured to separate an optical signal corresponding to a heartbeat from the received optical characteristic signal by subtracting a periodic signal common to the plurality of optical detectors.

56. The wearable system of claim 53, wherein: The at least one light emitter includes one or more of a photodiode and an LED.

57. The wearable system of claim 53, wherein: The plurality of optical detectors include photodetectors.

58. The wearable system of claim 53, further comprising a signal conditioner configured to modify the optical characteristic signal, the signal conditioner comprising one or more of a lens, a diffuser, a filter, and a lens array.

59. The wearable system of claim 53, wherein: The support member includes a garment.

60. The wearable system of claim 53, wherein: The support member may comprise a strap, a cuff, a patch or a belt.

61. The wearable system of claim 53, wherein: The support is configured to fit on one or more of a user's forearm, wrist, or hand.

62. The wearable system of claim 53, wherein: The processor is configured to output an indicator of muscle movement.

63. The wearable system of claim 62, wherein: The processor is configured to wirelessly output an indicator of the muscle movement.

64. The wearable system of claim 53, wherein: The processor is configured to associate each spectrophotometric sensor of the plurality of spectrophotometric sensors with one or more of muscle movement or body part movement.

65. The wearable system of claim 53, wherein: The processor includes processing circuitry.

66. The wearable system of claim 53, wherein: The processor comprises a remote processor.

67. A method of detecting position and / or movement of a body region, the method comprising: positioning a spectrophotometric sensor assembly on a skin region on a body of a subject, wherein the skin region is not part of the body region; collecting a spectrophotometric representation comprising a plurality of optical characteristic signals from the skin region using the set of spectrophotometric sensors; modifying the plurality of optical characteristic signals by separating one or more components of the optical characteristic signals corresponding to a heartbeat from the optical characteristic signals to form a modified spectrophotometric representation of the skin region; and An indicator of the position and / or movement of the body region is output based on the modified spectrophotometric representation.

68. The method of claim 67, further comprising transforming the modified spectrophotometric representation to account for one or more of: motion of the spectrophotometric sensor set relative to the skin region and shape changes of the shape of the skin region.

69. The method of claim 68, wherein: Transforming includes registering the modified spectrophotometric representation by comparing the modified spectrophotometric representation of the skin portion with a previous spectrophotometric representation of the skin portion and transforming the modified spectrophotometric representation based on the comparison.

70. The method of claim 67, further comprising determining the position and / or movement of the body region by correlating the spectrophotometric representation with previous positions and / or movements of the body region.

71. The method of claim 70, wherein: Correlating the spectrophotometric representation to the previous position and / or movement of the body region includes using a machine learning agent trained using multiple previous spectrophotometric representations of the skin and multiple video representations showing the region of the test subject's body at a time corresponding to each previous spectrophotometric representation in the multiple previous spectrophotometric representations.

72. The method of claim 70, wherein: Associating the spectrophotometric representation with the previous position and / or movement of the body region includes using a machine learning agent that is trained using multiple previous spectrophotometric representations of the skin region and multiple video representations that show the region of the body region at a time corresponding to each previous spectrophotometric representation in the multiple previous spectrophotometric representations.

73. The method of claim 67, wherein: The spectrophotometric representation was collected covering a 1 cm area from the skin 2 The multiple optical characteristic signals are collected in an area of ​​1000 square meters or larger.

74. The method of claim 73, wherein: Collecting includes collecting from a continuous or nearly continuous surface of the skin region.

75. The method of claim 67, wherein: Positioning the spectrophotometric sensor set includes positioning the spectrophotometric sensor set on a forearm to detect one or more of finger movement and position.

76. The method of claim 67, wherein: Positioning the set of spectrophotometric sensors includes wearing one or more of a garment, a strap, a cuff, a belt, or a patch.

77. The method of claim 67, wherein: Collecting the spectrophotometric representation includes emitting light of one or more wavelengths from one or more emitters of the spectrophotometric sensor group and detecting optical characteristic signals of the one or more wavelengths of light using one or more optical detectors of the spectrophotometric sensor group.

78. The method of claim 67, wherein: Outputting the indicator of the position and / or movement of the body region comprises indicating the position of the body region.

79. The method of claim 67, wherein: Outputting the indicator of the position and / or movement of the body region comprises triggering an effector based on a muscle movement.

80. The method of claim 67, wherein: Outputting the indicator of the position and / or movement of the body region includes transmitting the indicator to a remote processor.

81. A system for determining the position and / or movement of a body region, the system comprising: a spectrophotometric sensor set, the spectrophotometric sensor set comprising an array of spectrophotometric sensors, wherein each spectrophotometric sensor is configured to detect an optical property signal from the tissue, wherein the detected optical property signal corresponds to a spectrophotometric representation; a support configured to hold the spectrophotometric sensor adjacent to a skin surface; and a processor configured to: receive the spectrophotometric representation from the spectrophotometric sensor group, to adjust the alignment of the spectrophotometric representation by transforming the spectrophotometric representation based on a comparison between the spectrophotometric representation and a previous spectrophotometric representation, thereby forming an adjusted spectrophotometric representation of the region of the subject's skin, and to detect the position and / or movement of the body region based on the adjusted spectrophotometric representation after separating a component of the optical characteristic signal corresponding to the heartbeat from the received optical characteristic signal.

82. The system of claim 81, wherein: The anatomical features include skin ridges, vascular structures, tendons, or any combination of the skin ridges, vascular structures, and tendons.

Images