Method and apparatus for low-latency body state prediction based on neuromuscular data
Through the time-shifting neuromuscular activity signal and the multi-segment articulated rigid body system model, the waiting time and accuracy problems when predicting body state information are solved, and the user experience and application responsiveness are improved.
Patent Information
- Application Number
- CN202080025699.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Priority Date
- 2020-03-27
- Filing Date
- 2020-03-30
- Publication Date
- 2024-11-12
- Estimated Expiration
- 2040-03-30
AI Technical Summary
The prior art has high waiting times when predicting body state information, resulting in a reduced user experience, especially due to the time delay problem between neuromuscular activity signals and body part movement.
Through the time-shifting neuromuscular activity signal data relative to the body's state, inference models are trained to reduce waiting time, and a multi-segment articulated rigid body system is used to model the human musculoskeletal system, combining autonomous sensors such as IMU and neuromuscular sensors , generate more accurate predictions of body state.
Achieve lower body state waiting time and improved prediction accuracy, improving user experience, especially in artificial and virtual reality applications, and enhancing responsiveness to user movements.
Smart Images

Figure CN113632176B_ABST
Abstract
Description
[0001] CROSS-REFERENCE TO RELATED APPLICATIONS
[0002] This application claims priority to U.S. Application No. 62 / 826,516 filed on March 29, 2019, U.S. Application No. 62 / 841,054 filed on April 30, 2019, and U.S. Application No. 16 / 833,309 filed on March 27, 2020. The contents of U.S. Application No. 62 / 826,516, U.S. Application No. 62 / 841,054, and U.S. Application No. 16 / 833,309 are incorporated herein by reference in their entirety for all purposes. BRIEF DESCRIPTION OF THE DRAWINGS
[0004] The accompanying drawings illustrate many example embodiments and are a part of the specification. Together with the following description, these drawings demonstrate and explain various principles of the present disclosure.
[0005] Figure 1 is an illustration of an example block diagram of a system for predicting body state information according to an embodiment of the present disclosure.
[0006] Figure 2A is an illustration of an example graph depicting the effect of latency on predicted physical state information according to an embodiment of the present disclosure.
[0007] Figure 2B is an illustration of an example graph depicting latency reduction in predicted physical state information according to an embodiment of the disclosure.
[0008] Figure 3 is an illustration of an example graph depicting a relationship between a delay time interval and body state prediction accuracy according to an embodiment of the present disclosure.
[0009] Figure 4 Two graphs depicting user dependence in the relationship between delay time interval and body state prediction accuracy are shown according to embodiments of the present disclosure.
[0010] Figure 5 is an illustration of a flowchart of an example method for generating an inference model for predicting musculoskeletal position information using signals recorded from sensors according to an embodiment of the present disclosure.
[0011] Figure 6 is an illustration of a flowchart of an example method for determining body state information according to an embodiment of the present disclosure.
[0012] Figure 7 is an illustration of a perspective view of an example wearable device with a sensor according to an embodiment of the disclosure.
[0013] Figure 8 is an illustration of an example block diagram of a wearable device and a head mounted display according to an embodiment of the present disclosure.
[0014] Fig. 9 is an illustration of a flow chart of an example method for predicting body state based on neuromuscular data according to an embodiment of the disclosure.
[0015] Fig.10 is an illustration of example augmented reality glasses that may be used in conjunction with embodiments of the present disclosure.
[0016] Fig.11 is an illustration of an example virtual reality headset that may be used in conjunction with embodiments of the present disclosure.
[0017] Fig.12 is an illustration of an example haptic device that may be used in conjunction with embodiments of the present disclosure.
[0018] Fig.13 is an illustration of an example virtual reality environment according to an embodiment of the disclosure.
[0019] Fig.14 is an illustration of an example augmented reality environment according to an embodiment of the present disclosure.
[0020] In all drawings, the same reference symbols and descriptions indicate similar but not necessarily identical elements. Although the example embodiments described herein allow for various modifications and optional forms, specific embodiments are shown in the drawings as examples and will be described in detail herein. However, the example embodiments described herein are not intended to be limited to the disclosed specific forms. On the contrary, the present disclosure covers all modifications, equivalents, and substitutes that fall within the scope of the appended claims.
[0021] Detailed Description of Example Embodiments
[0022] The present disclosure generally relates to using trained inference models to predict the state of body parts of human users. In some computer applications that generate musculoskeletal representations of the human body, it may be desirable for the application to know the spatial location, orientation, and movement of the user's body to provide the application with a realistic representation of the body's movement. For example, in an artificial reality (AR) environment, tracking the spatial position of a user's hands can enable the application to accurately represent hand movements in the AR environment, which can allow the user to interact with virtual objects within the AR environment (e.g., by grasping or manipulating). In user interface applications, detecting the presence or absence of a user's pose or gesture can be used as a binary control input to a computer (e.g., mode switching). An important feature of a computer application that generates a musculoskeletal representation of the human body is the low latency between the movement of the user's body and the computer application's representation of the movement (e.g., displaying a visual representation to the user).
[0023] Depending on physiological differences between individuals and specific body parts, the time delay between the onset of neuromuscular activity (e.g., as indicated by an electromyogram (EMG) signal measured by a wearable device) and the contraction of a muscle in a body part can range from tens of milliseconds to hundreds of milliseconds or longer. Thus, at any point in time, the neuromuscular activity signal corresponds to a movement that may occur tens of milliseconds or more in the future.
[0024] The disclosed systems, methods, and apparatus for predicting the state of a body part or a portion of a body part based on neuromuscular activity data can achieve lower body state latency (e.g., latency from recorded neuromuscular data to the output of a trained inference model predicting the state of a user's body part or a portion of a body part) by temporally offsetting neuromuscular activity signal data relative to ground truth measurements of the body state. The temporally offset data set can be used as input for training an inference model and / or as input to a previously trained inference model.
[0025] In some embodiments, a method is provided, the method comprising: receiving neuromuscular activity signals in response to movement of a body part of a user via one or more neuromuscular sensors (e.g., neuromuscular sensors on a wearable device worn by the user), determining ground truth (e.g., directly observed) measurements associated with the corresponding movement of the body part of the user, time shifting the neuromuscular activity signals to substantially align with the timing of the corresponding movement, and training an inference model using the time-shifted neuromuscular activity signals.
[0026] All or part of the human musculoskeletal system can be modeled as a multi-segment articulated rigid body system, in which joints form interfaces between different segments, and joint angles define the spatial relationships between connected segments in the model. The limits of movement at the joints can be determined by the type of joints connecting the segments and the biological structures (e.g., muscles, tendons, ligaments, etc.) that limit the range of movement at the joints. For example, the shoulder joint connecting the upper arm and the trunk and the hip joint connecting the thigh and the trunk are ball and socket joints that allow extension and flexion movement as well as rotational movement. In contrast, the elbow joint connecting the upper arm and the forearm and the knee joint connecting the thigh and the calf allow a more limited range of motion. The musculoskeletal representation can be a multi-segment articulated rigid body system used to model part of the human musculoskeletal system. However, some segments of the human musculoskeletal system (e.g., the forearm), while approximated as rigid bodies in an articulated rigid body system, may include multiple rigid structures (e.g., the ulna and radius of the forearm) that provide more complex movement within the body segment that is not explicitly considered by the rigid body model. Therefore, the musculoskeletal representation may include body segments that represent combinations of body parts that are not strictly rigid bodies.
[0027] In some embodiments, the trained inference model can be configured to predict the state of a part of a user's body. Such a body state can include a force, movement, posture, or pose of a body part or a part of a body part. For example, the body state can include positional relationships between body segments in a musculoskeletal representation of the user's body part and / or force relationships of individual body segments and / or combinations of body segments.
[0028] The predicted force can be associated with one or more segments of the musculoskeletal representation of the user's body part. Such predicted force may include a linear force or a rotational (e.g., torque) force applied by one or more segments of the musculoskeletal representation. Examples of linear forces include, but are not limited to, the force of a finger or hand pressing on a solid object such as a table, or the force applied when two segments (e.g., two fingers) are squeezed together. Examples of rotational forces include, but are not limited to, the rotational force generated when the segments in the wrist and / or fingers twist and / or bend. In some embodiments, the predicted body state may include, but is not limited to, squeezing force information, pinching force information, gripping force information, torsion force information, bending force information, or information about the co-contraction force between muscles represented by the musculoskeletal representation.
[0029] The predicted movement can be associated with one or more segments of the musculoskeletal representation of the user's body part. Such predicted movement can include linear / angular velocity and / or linear / angular acceleration of one or more segments of the musculoskeletal representation. The linear velocity and / or angular velocity can be absolute (e.g., measured relative to a fixed reference frame) or relative (e.g., measured relative to a reference frame associated with another segment or body part).
[0030] As used herein, the term "gesture" may refer to a static configuration (e.g., positioning) of one or more body parts. For example, a gesture may include a fist, an open hand, an index finger statically pressed on a thumb, a palm pressed downward on a solid surface, a grip of a ball, or a combination thereof. As used herein, the term "gesture" may refer to a dynamic configuration of one or more body parts, the movement of one or more body parts, a force associated with a dynamic configuration, or a combination thereof. For example, a gesture may include waving a finger back and forth, throwing a ball, gripping a ball, or a combination thereof. Gestures and / or gestures may be defined by an application configured to prompt a user to perform gestures and / or gestures. Additionally or alternatively, gestures and / or gestures may be arbitrarily defined by a user.
[0031] In some embodiments, the body state can describe the user's hand, which can be modeled as a multi-segment articulated body. The joints in the wrist and each finger can form the joints between multiple segments in the model. In some embodiments, the body state can describe the combination of the hand and one or more arm segments of the user. The methods described herein are also applicable to musculoskeletal representations of body parts other than the hand (including but not limited to arms, legs, feet, torso, neck, or combinations thereof).
[0032] Compared to traditional methods, the systems and methods of compensating for electromechanical delays in the musculoskeletal system of the present disclosure can achieve lower latency and / or increased accuracy in predicting body states. Electromechanical delays in the musculoskeletal system can be defined as the time between the arrival of a motor neuron action potential at a neuromuscular synapse and the force output (e.g., movement) of a body part guided by the motor neuron action potential. The time delay between the onset of neuromuscular activity (e.g., indicated by an EMG signal from a wearable device worn by a user) and muscle contraction can range from tens of milliseconds to greater than hundreds of milliseconds, depending on the physiological condition of the user and the body part guided by the motor neuron action potential. Therefore, at any point in time, the EMG signal can correspond to a movement of a body part that occurs tens of milliseconds or more in the future.
[0033] In some examples, an inference model trained on neuromuscular signals that are temporally offset relative to ground truth measurements of body part states can evaluate the relationship between the neuromuscular signal and the corresponding movement of the body part, rather than the relationship between the neuromuscular signal and the movement corresponding to the earlier neuromuscular signal. In addition, the introduction of the temporal offset can reduce the latency between the ground truth body state and the predicted body state output by the trained inference model, thereby improving the user experience associated with an application (e.g., an artificial reality application, a user interface application, etc.) because the body part representation (e.g., a visual representation on a head mounted display) is more responsive to the user's actual motor control.
[0034] Electromechanical delays may vary between individuals and user body parts (e.g., hands and legs have different delays due to their different sizes). In some examples, the offset of neuromuscular signals relative to ground truth data about the position of the arm, hand, wrist, and / or fingers can be optimized based on specific physiology shared among users (e.g., age or gender), or personalized for specific users based on their individual electromechanical delays (e.g., for forearm muscles that control hand and finger movement). Training an inference model using neuromuscular signals that are offset in time relative to ground truth measurements of state can take into account any or all factors known to affect electromechanical delays in the human neuromuscular system, including but not limited to body temperature, fatigue, circadian cycle, medication consumption, diet, caffeine consumption, alcohol consumption, gender, age, flexibility, muscle contraction level, or a combination thereof.
[0035] In some examples, appropriate time offsets can be identified by generating multiple training data sets using multiple time offsets. In some examples, the time offsets can be different corresponding time intervals. For example, a time interval can be changed from 5ms to 100ms with an increment of 5ms or from 10ms to 150ms with an increment of 10ms, or a set of training data sets can be created with a start time interval, an end time interval, and some other combination of time increments. Multiple training data sets can be used to train multiple reasoning models. The waiting time and accuracy of these models can then be evaluated by comparing the model with basic fact data. A model showing an expected balance of waiting time and accuracy can be selected. The expected balance can depend on the task performed by the user. For example, a task that prioritizes precise movement (e.g., remote surgery) can accept longer waiting times in exchange for higher accuracy, while a task that prioritizes fast movement (e.g., video games) can accept lower accuracy in exchange for lower waiting times.
[0036] In some examples, an inference model trained with appropriate delay time intervals can be selected without generating multiple training data sets. For example, an inference model can be trained using known appropriate delay time intervals. The known appropriate delay time intervals can depend on known electromechanical delay times and / or known characteristic wait times of the system. The known electromechanical delay times can be specific to forces, movements, postures, postures, body parts, specific users, users with physiological characteristics (e.g., specific ages, genders, activity levels, or other characteristics that affect electromechanical delays in human neuromuscular systems), or a combination thereof. The known electromechanical delay times can be determined directly by a clinician for a specific user according to known methods, and / or estimated based on known electromechanical delay times for users who share physiological characteristics with the user.
[0037] In some examples, known electromechanical delay times for body parts, users, and / or user categories can be used to determine appropriate delay time intervals. For example, when the known electromechanical delay associated with a body part is 40ms, the time interval can be selected to be in the range of 20ms to 60ms. Prediction accuracy can be generated for an inference model trained using a time-shifted training data set, which is generated using a selected time interval. The generated prediction accuracy can be used to select one or more inference models to predict body part states. By selecting time intervals based on known electromechanical delay times, the selection of appropriate delay time intervals can focus on time intervals that may combine sufficient accuracy and low latency. As a result, fewer time intervals can be tested and / or a range of time intervals can be tested at a higher resolution (e.g., a resolution of 1ms instead of a resolution of 5ms or 10ms).
[0038] Figure 1 A system 100 according to an embodiment of the present disclosure is shown. The system 100 may include a plurality of sensors 102 configured to record signals generated according to the movement of a human body part. The sensor 102 may include an autonomous sensor. In some embodiments, the term "autonomous sensor" may refer to a sensor configured to measure the movement of a body segment without the need for an external device. In additional embodiments, the sensor 102 may also include a non-autonomous sensor combined with an autonomous sensor. In some examples, the term "non-autonomous sensor" may refer to a sensor configured to measure the movement of a body segment using an external device. Examples of non-autonomous sensors may include, but are not limited to, wearable (e.g., mounted on the body) cameras, global positioning systems, laser scanning systems, radar ranging sensors, or combinations thereof.
[0039] Autonomous sensors may include multiple neuromuscular sensors configured to record signals generated according to neuromuscular activity in human muscles. The term "neuromuscular activity" used herein may refer to neural activation of spinal motor neurons that control muscles, muscle activation, muscle contraction, or any combination thereof. Neuromuscular sensors may include one or more electromyography (EMG) sensors, one or more myography (MMG) sensors, one or more sonomyography (SMG) sensors, one or more sensors of any suitable type configured to detect neuromuscular signals, or a combination thereof. In some examples, sensor 102 may be used to sense muscle activity associated with movement of a body part controlled by a muscle. Sensor 102 may be configured and arranged to sense muscle activity. Spatial information (e.g., position and / or orientation information) and force information describing movement may be predicted based on the sensed neuromuscular signals as the user moves over time.
[0040] Autonomous sensors may include one or more inertial measurement units (IMUs) that may measure a combination of physical aspects of motion using, for example, accelerometers, gyroscopes, magnetometers, or combinations thereof. In some examples, an IMU may be used to sense information about the movement of a body part to which the IMU is attached, and information derived from the sensed data (e.g., position and / or orientation information) may be tracked as the user moves over time. For example, one or more IMUs may be used to track the movement of a user's body part (e.g., arm, leg) proximal to the user's torso as the user moves over time.
[0041] Some embodiments may include at least one IMU and multiple neuromuscular sensors. The IMU and neuromuscular sensors may be arranged to detect movement of different parts of the human body. For example, the IMU may be arranged to detect movement of one or more body segments (e.g., upper arms) at the proximal end of the trunk, and the neuromuscular sensors may be arranged to detect movement of one or more body segments (e.g., forearms or wrists) at the distal end of the trunk. Autonomous sensors may be arranged in any suitable manner, and embodiments of the present disclosure are not limited to any particular sensor arrangement. For example, at least one IMU and multiple neuromuscular sensors may be co-located on a body segment to track the movement of the body segment using different types of measurements. In some examples, the IMU sensor and multiple EMG sensors may be arranged on a wearable device configured to be worn around the user's lower arm (e.g., forearm) or wrist. In this arrangement, the IMU sensor may be configured to track movement information (e.g., position, velocity, acceleration, and / or orientation over time) associated with one or more arm segments. Movement information may determine, for example, whether the user has raised or lowered their arm. The EMG sensor may be configured to determine movement information associated with the wrist or hand segments to determine, for example, whether the user has an open or closed hand configuration.
[0042] Each autonomous sensor may include one or more sensing components configured to sense information about the user. In the case of an IMU, the sensing component may include one or more accelerometers, gyroscopes, magnetometers, or any combination thereof to measure characteristics of body motion. Examples of characteristics of body motion may include, but are not limited to, acceleration, angular velocity, linear velocity, and sensed magnetic fields around the body. The sensing components of the neuromuscular sensor may include, but are not limited to, electrodes configured to detect electrical potentials on the surface of the body (e.g., for an EMG sensor), a vibration sensor configured to measure vibrations of the skin surface (e.g., for an MMG sensor), an acoustic sensing component configured to measure ultrasonic signals generated by muscle activity (e.g., for an SMG sensor), or a combination thereof.
[0043] In some examples, hardware signal processing circuitry can be used to process the output of sensor 102 (e.g., to perform amplification, filtering, and / or rectification). In some examples, at least some signal processing of the output of sensor 102 can be performed in software. Thus, signal processing of autonomous signals recorded by autonomous sensors can be performed in hardware, software, or any suitable combination of hardware and software, as embodiments of the present disclosure are not limited in this respect.
[0044] In some examples, as described in more detail below, the recorded sensor data from the sensor 102 can be processed to calculate additional derived measurements that are provided as inputs to the inference model 104. For example, the recorded signals from the IMU sensor can be processed to derive an orientation signal that specifies the orientation of the rigid body segment over time. The autonomous sensor can implement the signal processing using components that are integrated with the sensing components, or a portion of the signal processing can be performed by one or more components that communicate with but are not directly integrated with the sensing components of the autonomous sensor.
[0045] In some examples, multiple autonomous sensors may be arranged as part of a wearable device that is configured to be worn (e.g., don) on or around a user's body part. For example, an IMU sensor and / or multiple neuromuscular sensors may be arranged circumferentially around an adjustable and / or elastic band, such as a wristband or armband configured to be worn around a user's wrist or arm. In some examples, an IMU sensor and / or multiple neuromuscular sensors may be arranged and / or attached to a part and / or multiple parts of the body, including but not limited to an ankle, waist, torso, neck, head, foot, shin, shoulder, or a combination thereof. Additionally or alternatively, an autonomous sensor may be arranged on a wearable patch that is configured to be attached to a part of the user's body. In some examples, multiple wearable devices may be used to predict musculoskeletal position information involving movement of multiple parts of the body, each wearable device including one or more IMUs and / or neuromuscular sensors thereon.
[0046] In some examples, sensor 102 may include only a plurality of neuromuscular sensors (e.g., EMG sensors). In some examples, sensor 102 may include a plurality of neuromuscular sensors and at least one "auxiliary" or additional sensor configured to continuously record a plurality of auxiliary signals. Examples of auxiliary sensors may include, but are not limited to, other autonomous sensors (such as IMU sensors), non-autonomous sensors (such as imaging devices (e.g., cameras), radar ranging sensors, radiation-based sensors, laser scanning devices), and / or other types of sensors (such as heart rate monitors).
[0047] The system 100 may also include at least one processor 101 programmed to communicate with the sensors 102. For example, signals recorded by one or more sensors 102 may be provided to the processor 101, which may be programmed to execute one or more machine learning algorithms that process the signals output by the sensors 102 to train one or more inference models 104. The trained (or retrained) inference models 104 may be stored for later use in generating musculoskeletal representations 106, as described in more detail below. Non-limiting examples of inference models 104 that may be used to predict body state information based on signals recorded from the sensors 102 are discussed in detail below.
[0048] The system 100 may include a display device 108 configured to display a visual representation of a body state (e.g., a visual representation of a hand). As discussed in more detail below, the processor 101 may use one or more trained reasoning models 104 that are configured to predict body state information based at least in part on signals recorded by the sensor 102. The predicted body state information may be used to update a musculoskeletal representation 106, which may be used to render a visual representation on a display device 108 (e.g., a head-mounted display). Real-time reconstruction of the current body state and subsequent rendering of a visual representation reflecting the current body state information in the musculoskeletal model on the display device 108 may provide the user with visual feedback on the effectiveness of the reasoning model 104 in accurately representing the expected body state. In some examples, metrics associated with the musculoskeletal representation 106 (e.g., likelihood metrics or quality metrics of one or more gestures, the quality metrics representing the confidence level of estimating the position, movement, and / or force of a segment of a multi-segment articulated rigid body system (such as a hand)) may be provided to the user or other third party.
[0049] In some examples, a computer application configured to simulate an artificial reality environment can be instructed to display a visual representation of a user's hand on display device 108. Positioning, movement, and / or forces exerted by portions of the hand within the artificial reality environment can be displayed based on the output of the trained reasoning model. As signals are continuously recorded by sensor 102 and processed by trained reasoning model 104, the visual representation of the user's position, movement, and / or force can be dynamically (e.g., in real time) updated based on the currently reconstructed body state information.
[0050] As described above, some embodiments may involve using an inference model 104 to predict a musculoskeletal representation 106 based on signals recorded from sensors 102 (e.g., wearable autonomous sensors). The inference model 104 can be used to predict musculoskeletal position information without having to place sensors 102 on every segment of a rigid body to be represented in the computer-generated musculoskeletal representation 106. The types of joints between segments in a multi-segment articulated rigid body model can constrain the movement of the rigid body. Additionally, different users may tend to move in various ways when performing tasks that can be captured in the statistical patterns of individual user movements. At least some of these constraints on human movement can be explicitly incorporated into the inference model 104 used for prediction. Additionally or alternatively, the constraints can be learned by the inference model 104 through training based on data recorded from the sensors 102. The constraints imposed on the construction of the inference model 104 can be constraints determined by the anatomy and physics of the user's body, while the constraints derived from the statistical patterns can be constraints determined by the human behavior of one or more users from whom the sensor measurements are recorded.
[0051] As discussed above, some embodiments may involve using the inference model 104 to predict body state information to enable generation and / or real-time updating of the computer-based musculoskeletal representation 106. When the user performs one or more movements, the inference model 104 may be used to predict body state information based on signals from the sensors 102, including but not limited to IMU signals, neuromuscular signals (e.g., EMG signals, MMG signals, and SMG signals), external device signals (e.g., camera, radar, or laser scanning signals), or combinations thereof.
[0052] Figure 2A An example graph depicting the effect of latency on predicted body state information is shown in accordance with an embodiment of the present disclosure. The system may be configured to obtain repeated (e.g., periodic) measurements of neuromuscular signals 203 and body state 201 (e.g., ground truth body state) as a user performs one or more movements. For example, neuromuscular signals 203 and ground truth body state 201 may be time series data (e.g., data recorded over a period of time), including measurements that are explicitly and / or implicitly timestamped (e.g., a tuple of a measurement value and a measurement time, and / or a sequence of measurements with a known sampling time interval and a known start time). The system may be configured to align samples of body state 201 and signal 203 based on acquisition time. Alignment of samples of body state 201 and signal 203 may involve upsampling, downsampling, interpolation, other signal processing techniques, or a combination thereof. For example, as Figure 2A As shown in FIG. , the system can make the body state sample {B T0 ,B T0+Δt,B T0+2Δt ,B T0+3Δt ,B T0+4Δt ,…} and signal samples {S T0 ,S T0+Δt ,S T0+2Δt ,S T0+3Δt ,S T0+4Δt ,…} are aligned respectively.
[0053] The system can be configured to train an inference model using body state 201 as ground truth data for signal 203. In some examples, the term "ground truth data" can be used interchangeably with the term "label time series data". Label time series data can be data collected at constant or variable time intervals over a period of time. A conventional system can be configured to use a current signal sample to predict a current body state sample (e.g., as shown by arrow 202 connecting a signal sample to a body state at the same time, based on the Figure 2A S T0 Prediction B T0 ). Due to electromechanical delay, body state B T0+Δt may be the result of previous muscle activity. Therefore, earlier signal samples (e.g., S T0 ) More accurate prediction of physical status B T0+Δt . In addition, predicting the body state from the signal samples requires processing time. This processing time can include time delays associated with: time integration of the signal, signal recording and conditioning, signal data transmission (e.g., from a wearable sensor to a processing system), memory access, processor instruction execution, and processing the signal data using an inference model. Such time delays can range between 10ms and 100ms (or longer).
[0054] Predicted body state 205 may depict when a sample generated using signal 203 is output by a trained inference model (as shown by arrow 206 connecting the sample of signal 203 to predicted body state 205). Figure 2A As shown, when the trained inference model outputs the predicted body state B T0 When the most recently measured body part state can be B T0+Δt As used herein, a waiting time may be a period of time (e.g., an average period of time, a median period of time, or other suitable period of time) between the measurement of a body state and the output of a corresponding predicted body state 205 (e.g., the measured body state B T0 and predicted physical state B T0The waiting time may degrade the quality of the user experience because the user may feel that the output of the system (e.g., a visual representation of the body state displayed on a head mounted display (HMD)) lags behind the user's actual movement.
[0055] Figure 2B A graph depicting the effect of training an inference model using time-shifted training data on latency 217 is shown in accordance with an embodiment of the present disclosure. Figure 2A As described, the system may obtain multiple samples of body state 211 (e.g., ground truth body state) and signal 213. In some examples, the system may be configured to pair a sample of signal 213 with a sample of body state 211 acquired later (as indicated by arrow 212 connecting the sample of signal 213 with the sample of body state 211), rather than pairing a sample of signal 213 with a sample of body state 211 acquired simultaneously. For example, the system may pair a sample of signal S T0 Sample B with physical condition T0+Δt In this way, the system can create a training data set by time-shifting the signal 213 or the basic truth body state 211. The system can be configured to train the inference model using the time-shifted training data set. For example, the inference model can then be trained using the time-shifted training data set to predict the body state 211 based on the signal 213.
[0056] Predicted body state 215 depicts when a sample generated using signal 213 is output by the trained inference model (as shown by arrow 216 connecting the sample of signal 213 and predicted body state 215). In this example, when the trained inference model outputs predicted body state B T0+Δt When the state of the body part measured most recently is also B T0+Δt As shown in the figure, by T0 Prediction B T0+Δt ,and Figure 2A Compared with the waiting time 207 shown, the waiting time 207 can be reduced in the physical state B. T0+Δt When it appears, it is the same as the predicted body state B output by the trained inference model T0+Δt As discussed herein, the inference model can be trained to calculate the waiting time between S T0 Prediction B T0+Δt This is at least in part because electromechanical delays cause the signal measured at time T0 to affect the body state that occurs later (e.g., at T 0+Δt Therefore, for an appropriate choice of the delay time interval Δt, the inference model is trained to predict the body state according to S T0 Prediction B T0+Δt It can improve the accuracy of physical condition prediction. Figure 3 and Figure 4 Example methods of selecting the delay time interval Δt are discussed.
[0057] Figure 3 A graph 300 depicting an empirical relationship between latency time interval Δt and body state prediction accuracy according to an embodiment of the present disclosure is shown. The empirical relationship can be used to select a trained inference model that exhibits a desired balance between latency and body state prediction accuracy. Figure 3 The independent variable depicted in is the delay time interval between the neuromuscular signal sample and the body state sample. Positive time interval values correspond to pairing the neuromuscular signal sample with a body state sample obtained after the neuromuscular signal sample. Negative time interval values correspond to pairing the neuromuscular signal sample with a body state sample obtained before the neuromuscular signal sample. A zero time interval (0.0 ms) value corresponds to pairing the signal sample with a body state sample obtained at the same time as the signal sample. Figure 3 The response variable depicted in the graph of can be a measure of the prediction accuracy of the model trained using the training data set with a time shifted time interval. The depicted measure can be a correlation value between the measured and predicted joint angles in the musculoskeletal representation of the hand. In some examples, other measures of prediction accuracy can be used, such as the mean square error between the characteristic values of the musculoskeletal representation of the body part. These characteristic values can include, but are not limited to, the joint angles, forces, or spatial coordinates of the body part. Similarly, the possibility of correctly predicting a known posture or posture (e.g., a fist posture or a transition from an open hand to a fist posture) can be used as a measure of prediction accuracy. For example, the body part state and the predicted body part state can be a binary label indicating the presence or absence of a posture or posture. The trained inference model may have a false positive, false negative, true positive, or true negative prediction rate. The measure of prediction accuracy can depend on at least one of these prediction rates.
[0058] As shown in the graph 300, as the delay time interval value increases from zero to 20 milliseconds, the body state prediction accuracy (e.g., the correlation between the measured and predicted joint angles) can be improved. Thereafter, as the delay time interval value increases, the prediction accuracy decreases. As shown, offsetting the measured signal by 40ms relative to the body state label reduces the waiting time without reducing the prediction accuracy. As described herein, depending on the task, an inference model trained using a shorter or longer time interval (e.g., a time interval in the range of 10ms to 100ms) can be selected for use in predicting body state.
[0059] In some examples, an inference model can be selected for predicting a body state based on a prediction accuracy criterion (e.g., a correlation between measured and predicted joint angles) and a latency time interval Δt of a training data set used to generate the training inference model. For example, among the inference models that meet the prediction accuracy criterion (e.g., an accuracy higher than a set threshold), the selected inference model can be an inference model trained using a training data set that is generated using a maximum time interval. For example, two inference models can meet the accuracy criterion (e.g., the accuracy of both models is higher than an acceptable threshold). The first model can have a higher accuracy than the second model, but the time interval used to generate the training data set for training the first model can be less than the time interval used to generate the training data set for training the second model. In this example, the second inference model can be selected to predict the body state because the second inference model can have an acceptable prediction accuracy and a lower latency than the first inference model.
[0060] The accuracy criterion may depend on the maximum accuracy observed between the inference models. For example, the accuracy criterion may be expressed as a deviation from the accuracy of the most accurate model. An inference model may meet the accuracy criterion when the deviation of the accuracy of the inference model is less than a threshold. The threshold may be an absolute difference in accuracy (e.g., the most accurate model has a prediction accuracy of 85% and the second model has an accuracy of at least 80%). Alternatively, the threshold may be a relative difference in accuracy (e.g., a less accurate model has an accuracy of at least 95% of the most accurate model).
[0061] Figure 4 Two graphs depicting user dependence in the empirical relationship between time interval and prediction accuracy according to an embodiment of the present disclosure are shown. The dependence of prediction accuracy on delay time interval may vary from user to user. Figure 4 As shown in the graphs of , the dependence of prediction accuracy on the delay time interval can vary between user A shown in graph 402 and user B shown in graph 404. Therefore, the system can be customized for the user by selecting an inference model trained using a delay time interval appropriate for the user and / or training the inference model using a training data set generated with delay time intervals appropriate for the user. The appropriate delay time interval can depend on known electromechanical delay times and / or characteristic waiting times of the system. For example, user A and user B can have different electromechanical delay times based on physiological characteristics (e.g., user age, gender, activity level, or other characteristics known to affect electromechanical delays in the human neuromuscular system).
[0062] Figure 5A method 500 for generating (e.g., training) an inference model using signals recorded from a sensor (e.g., sensor 102) is described. The method 500 may be performed using any suitable computing device, as the embodiments of the present disclosure are not limited in this respect. For example, the method 500 may be performed by reference to Figure 1 and Figure 7 As another example, one or more operations of method 500 may be performed using one or more servers (e.g., servers included as part of a cloud computing environment). For example, at least a portion of the operations in method 500 may be performed using a cloud computing environment and / or a wearable device (such as Figure 7 Wearable devices 700, Figure 8 Wearable devices 810, Fig.11 Wearable devices 1100, Fig.12 Wearable devices 1200, Fig.13 Wearable devices 1320, Fig.14 The operation of method 500 is performed by a processor of the wearable device 1404. Figure 5 Although shown as being performed in a certain order, the operations of method 500 may be performed in any order.
[0063] Method 500 may include operation 502, in which a plurality of sensor signals (e.g., neuromuscular signals, IMU signals, etc.) are obtained for one or more users performing one or more movements (e.g., playing an artificial reality game). In some examples, the plurality of sensor signals may be recorded as part of method 500. Additionally or alternatively, the plurality of sensor signals may have been recorded prior to execution of method 500 and accessed (rather than recorded) at operation 502.
[0064] In some examples, the plurality of sensor signals may include sensor signals recorded for a single user performing a single movement and / or multiple movements. A user may be instructed to perform a series of movements for a particular task (e.g., grasping a game controller, providing user input to a computer, etc.), and sensor signals corresponding to the user's movements may be recorded as the user performs the task the user is instructed to perform. The sensor signals may be recorded by any suitable number and / or type of sensors located at any suitable location to detect user movements associated with the task being performed. For example, after a user is instructed to perform a task with the fingers of the user's right hand, the sensor signals may be recorded by a plurality of neuromuscular sensors and one or more IMU sensors arranged around the user's right lower arm (e.g., circumferentially) to detect muscle activity in the right lower arm that causes the right hand to move, and the one or more IMU sensors are arranged to predict the joint angles of the user's arm relative to the user's torso. As another example, after a user is instructed to perform a task with the user's legs (e.g., kicking an object), sensor signals may be recorded by multiple neuromuscular sensors and one or more IMU sensors that are arranged around the user's legs (e.g., circumferentially) to detect muscle activity in the legs that causes the foot to move, and the one or more IMU sensors that are arranged to predict the joint angles of the user's legs relative to the user's torso.
[0065] In some examples, the sensor signals obtained in operation 502 may correspond to signals from one type of sensor (e.g., one or more IMU sensors or one or more neuromuscular sensors), and the inference model may be trained based on the sensor signals recorded using the particular type of sensor, thereby producing a trained inference model specific to the sensor type. For example, the obtained sensor signals may include multiple EMG sensor signals arranged around a user's lower arm or wrist (e.g., circumferentially), and the inference model may be trained to predict musculoskeletal position information for movement of the wrist and / or hand during performance of a task such as grasping and twisting an object (such as a game controller or door handle).
[0066] In embodiments where predictions are provided based on multiple types of sensors (e.g., IMU sensors, EMG sensors, MMG sensors, SMG sensors, etc.), separate inference models may be trained for each of the different types of sensors, and the outputs of the sensor type-specific models may be combined to generate a musculoskeletal representation of the user's body. In some examples, sensor signals obtained from two or more different types of sensors in operation 502 may be provided to a single inference model that is trained based on signals recorded from the different types of sensors. For example, as discussed in more detail below, an IMU sensor and multiple EMG sensors may be arranged on a wearable device configured to be worn around a user's forearm, and the signals recorded by the IMU sensor and the EMG sensor may be provided jointly as input to the inference model.
[0067] In some examples, a user may be instructed to perform a task multiple times, and sensor signals and position information may be recorded for each of the user's multiple task repetitions. In some examples, multiple sensor signals may include signals recorded for multiple users, each of which performs the same task once or more times. Each of the multiple users may be instructed to perform a task, and when the user performs the task in accordance with the instructions (once or repeatedly), sensor signals and position information corresponding to the user's movements may be recorded. When sensor signals are collected from multiple users and these sensor signals are combined to generate an inference model, it can be assumed that different users use similar musculoskeletal positions to perform the same movement. Collecting sensor signals and position information from a single user who repeatedly performs the same task and / or from multiple users who perform the same task once or more facilitates collecting sufficient training data to generate an inference model that can accurately predict musculoskeletal position information associated with the performance of the task.
[0068] In some examples, a user-independent inference model can be generated based on training data corresponding to recorded signals from multiple users, and when the system is used by the user, the inference model can be trained based on the recorded sensor data so that the inference model learns user-related features to refine the system's predictive capabilities for specific users and improve prediction accuracy.
[0069] In some examples, the multiple sensor signals can include signals recorded for a user (or each of the multiple users) performing each of the multiple tasks one or more times. For example, a user can be instructed to perform each of the multiple tasks (e.g., grasping an object, pushing an object, pulling open a door, etc.), and as the user performs each of the multiple tasks that the user was instructed to perform, signals corresponding to the user's movements can be recorded. Collecting such signal data can facilitate the development of an inference model for predicting musculoskeletal position information associated with multiple different actions that the user may perform. For example, training data containing musculoskeletal position information for multiple actions can facilitate the generation of an inference model for predicting which of multiple possible movements a user may be performing.
[0070] As discussed above, the sensor data obtained in operation 502 can be obtained by recording sensor signals when each of one or more users performs each of one or more tasks one or more times. In operation 504, basic fact data (e.g., labeled time series data) can be obtained by multiple sensors, including but not limited to optical sensors, inertial measurement sensors, mutual magnetic induction measurement sensors, pressure sensors, or combinations thereof. The basic fact data can indicate the state of the user's body parts. For example, when the user performs a task, position information describing the spatial positions of different body segments during the task performance can be obtained in operation 504. In some examples, the position information can be obtained using one or more external devices or systems that track the positions of different points on the body during the task performance. For example, a motion capture system, a laser scanner, a device for measuring mutual magnetic induction, some other system configured to capture position information, or a combination thereof can be used. As a non-limiting example, multiple position sensors can be placed on segments of fingers of a user's hand, and when the user performs a task (e.g., grasping an object), the motion capture system can be used to determine the spatial position of each position sensor. Additionally or alternatively, neuromuscular signals may be obtained at operation 502 and may be used alone or in combination with one or more images or IMU signals from a motion capture system to determine the spatial position of a user's body part (e.g., finger) as the user performs a task. The sensor data obtained at operation 502 may be recorded simultaneously with the recording of the position information obtained at operation 504. In this example, position information indicating the position of each finger segment over time when performing a grasping motion is obtained.
[0071] Method 500 may proceed to operation 506, where the sensor signals obtained in operation 502 and / or the position information obtained in operation 504 are optionally processed. For example, the sensor signals and / or position information signals may be processed using, but not limited to, amplification, filtering, rectification, other types of signal processing, or combinations thereof.
[0072] Method 500 may proceed to operation 508, where musculoskeletal position characteristics are determined based on position information, such as the position information collected in operation 504. In some examples, rather than using recorded spatial (e.g., x, y, z) coordinates corresponding to a position sensor as training data to train an inference model, a set of derived musculoskeletal position characteristic values is determined based on the recorded position information, and the derived values are used as training data to train the inference model. For example, using information about constraints between connected pairs of rigid segments in an articulated rigid body model, the position information may be used to determine the joint angles between each connected pair of rigid segments at each of a plurality of time points during task performance. Accordingly, the position information obtained in operation 504 may be represented by a vector of n joint angles at each of a plurality of time points, where n is the number of joints or connections between segments in the articulated rigid body model.
[0073] Method 500 may proceed to operation 510, where the time series information obtained in operations 502 and 508 may be combined to create training data for training an inference model. The obtained data may be combined using any suitable method. In some examples, each sensor signal obtained in operation 502 may be associated with a task or movement within a task that corresponds to a musculoskeletal position characteristic (e.g., joint angle) determined based on the position information obtained in operation 504 when the user performs the task or movement. In this way, sensor signals may be associated with musculoskeletal position characteristics (e.g., joint angles), and an inference model may be trained to predict that when specific sensor signals are recorded during the performance of a specific task, the musculoskeletal representation will be characterized by specific musculoskeletal position characteristics between different body segments.
[0074] In embodiments including different types of sensors (e.g., IMU sensors and neuromuscular sensors) configured to simultaneously record different types of movement information (e.g., position information, velocity information, acceleration information) during task execution, sensor data of different types of sensors may be recorded using the same or different sampling rates. When sensor data is recorded at different sampling rates, at least some of the sensor data may be resampled (e.g., upsampled or downsampled) so that all sensor data provided as input to the inference model corresponds to time series data of the same time resolution (e.g., time period between samples). At least some of the sensor data may be resampled using any suitable method (including, but not limited to, using interpolation for upsampled sensor data and decimation for downsampled sensor data).
[0075] Additionally or alternatively, some embodiments may employ an inference model configured to accept multiple inputs asynchronously. For example, the inference model may be configured to model the distribution of "missing" values in input data having a lower sampling rate. Additionally or alternatively, the training timing of the inference model may occur asynchronously when inputs from multiple sensor data measurements become available as training data (e.g., after signal conditioning).
[0076] Combining the time series information obtained in operations 502 and 508 to create training data for training the inference model in operation 510 may include generating one or more training data sets. As described herein, the one or more training data sets may be generated by time-shifting the sensor signals obtained in operation 502 or by time-shifting the ground truth data obtained in operation 504 or operation 508 by one or more time intervals.
[0077] Method 500 may proceed to operation 512, where an inference model for predicting musculoskeletal position information may be trained using the training data generated in operation 510. The trained inference model may use a sequence of data sets as input, and each data set in the sequence may include an n-dimensional vector of sensor data. The inference model may provide an output indicating the likelihood that, for each of one or more tasks or movements that may be performed by the user, a musculoskeletal representation of the user's body will be characterized by a set of musculoskeletal position characteristics (e.g., a set of joint angles between segments in an articulated multi-segment body model). For example, the inference model may use as input a sequence of vectors {xk|1≤k≤K} generated using measurements obtained at time points t1, t2, ..., tK, where the i-th component of the vector xj may be a value measured by the i-th sensor at time tj and / or a value derived from a value measured by the i-th sensor at time tj. In another non-limiting example, the derived values provided as input to the inference model may include features extracted from data from all sensors or a subset of sensors at time tj and / or before time tj (e.g., a covariance matrix, a power spectrum, any other suitable derived representation, or a combination thereof). Based on such input, the inference model may provide an output indicating the probability that a musculoskeletal representation of the user's body will be characterized by a set of musculoskeletal position characteristics. As a non-limiting example, the inference model may be trained to predict a set of joint angles for segments in the fingers of a hand over time as the user grasps an object. In this example, the trained inference model may output a set of predicted joint angles for the joints in the hand corresponding to the sensor input.
[0078] In some examples, the inference model may be a neural network. In some examples, the inference model may be a recurrent neural network. The recurrent neural network may be a long short-term memory (LSTM) neural network. However, the recurrent neural network is not limited to the LSTM neural network, and may have any other suitable architecture. For example, the recurrent neural network may be, but is not limited to, a fully recurrent neural network, a recursive neural network, a variational autoencoder, a Hopfield neural network, an associative memory neural network, an Elman neural network, a Jordan neural network, an echo state neural network, a second-order recursive neural network, any other suitable type of recurrent neural network, or a combination thereof. In some examples, a neural network that is not a recurrent neural network may be used. For example, a deep neural network, a convolutional neural network, a feedforward neural network, or a combination thereof may be used.
[0079] In some examples where the inference model is a neural network, an output layer of the neural network can provide a set of output values corresponding to a corresponding set of possible musculoskeletal position characteristics (e.g., joint angles). In this example, the neural network can operate as a nonlinear regression model that is configured to predict musculoskeletal position characteristics from raw and / or processed (e.g., conditioned) sensor measurements. In some examples, other suitable nonlinear regression models can be used in place of the neural network, as the present disclosure is not limited in this respect.
[0080] In some examples, the neural network can be implemented based on multiple and / or different types of topologies and / or architectures, including deep neural networks with fully connected (dense) layers, long short-term memory (LSTM) layers, convolutional layers, temporal convolutional layers (TCL), other suitable types of deep neural network topologies and / or architectures, or combinations thereof. The neural network can have different types of output layers, including but not limited to output layers with logistic sigmoid activation functions, hyperbolic tangent activation functions, linear units, rectified linear units, other suitable types of nonlinear units, or combinations thereof. In some examples, the neural network can be configured to represent probability distributions over n different categories via a softmax function. In some examples, the neural network can include an output layer that provides a parameterized distribution (e.g., the mean and / or variance of a Gaussian distribution).
[0081] Embodiments of the present disclosure are not limited to the use of neural networks, as other types of inference models may be employed. In some examples, the inference model may include, but is not limited to, a hidden Markov model, a Markov switching model that allows switching between different dynamic systems, a dynamic Bayesian network, any other suitable graphical model with a time component, or a combination thereof. At operation 512, any such inference model may be trained using the sensor data obtained at operation 502.
[0082] As another example, the inference model may use features derived from the sensor data obtained in operation 502 as input. In such an embodiment, in operation 512, the inference model may be trained using features extracted from the sensor data obtained in operation 502. The inference model may include, but is not limited to, a support vector machine, a Gaussian mixture model, a regression-based classifier, a decision tree classifier, a Bayesian classifier, any other suitable classifier, or a combination thereof. Any suitable method may be used to derive input features to be provided to the inference model as training data from the sensor data obtained in operation 502. For example, the sensor data may be analyzed as time series data using, but not limited to, wavelet analysis techniques (e.g., continuous wavelet transform, discrete time wavelet transform, etc.), Fourier analysis techniques (e.g., short-time Fourier transform, discrete time Fourier transform, Fourier transform, etc.), any other suitable type of time-frequency analysis techniques, or a combination thereof. As a non-limiting example, the sensor data may be transformed using a wavelet transform, and the resulting wavelet coefficients may be provided as input to the inference model.
[0083] In some examples, at operation 512, parameter values of the inference model can be estimated based on the training data generated at operation 510. For example, when the inference model is a neural network, the parameters (e.g., weights) of the neural network can be estimated based on the training data. The parameters of the inference model can be estimated using, but not limited to, gradient descent, stochastic gradient descent, any other suitable iterative optimization technique, or a combination thereof. In an embodiment where the inference model is a recurrent neural network (e.g., an LSTM neural network), the inference model can be trained using stochastic gradient descent and backpropagation through time. The training can employ a cross-entropy loss function and / or any other suitable loss function, as the present disclosure is not limited in this respect.
[0084] The method 500 may proceed to operation 514, where the trained inference model may be stored (e.g., in a data storage, a local database, a remote cloud database, a memory, etc.). The trained inference model may be stored using any suitable format, device, and / or method. In this manner, the inference model generated during execution of the method 500 may be used later. For example, as described below, a state prediction system may be configured using the trained inference model to predict body part states based on neuromuscular activity time series data (e.g., predict musculoskeletal position information, such as joint angles, based on a given set of input sensor data).
[0085] In some examples, sensor signals can be recorded from a plurality of sensors (e.g., disposed on or near a user's body surface) that record activity associated with body movement during task performance. The recorded signals can optionally be processed (e.g., conditioned) and provided as input to a computer program product using the above combination of Figure 5 An inference model trained using one or more of the techniques described herein. In some examples, autonomous signals can be recorded continuously, and the continuously recorded (raw or processed) signals can be continuously and / or periodically provided as input to a trained inference model for predicting musculoskeletal position information (e.g., joint angles) for a given set of input sensor data. As discussed above, in some examples, the trained inference model can be a user-independent model trained based on autonomous sensors and position information measurements from multiple users. In some examples, the trained model can be a user-dependent model trained based on data recorded from an individual user, and the data associated with the sensor signals is also collected from the individual user.
[0086] After the trained inference model receives the sensor data as a set of input parameters, predicted musculoskeletal position information can be output from the trained inference model. As discussed above, in some examples, the predicted musculoskeletal position information can include a set of musculoskeletal position information values (e.g., a set of joint angles) for a multi-segment articulated rigid body model representing at least a portion of the user's body. In some examples, the musculoskeletal position information can include a set of probabilities that the user is performing one or more moves in a set of possible moves.
[0087] In some examples, after predicting the musculoskeletal position information, a computer-based musculoskeletal representation of the user's body can be generated based at least in part on the musculoskeletal position information output from the trained inference model. The computer-based musculoskeletal representation can be generated using any suitable method. For example, a computer-based musculoskeletal model of the human body can include a plurality of rigid body segments, each of which corresponds to one or more bone structures in the body. For example, the upper arm can be represented by a first rigid body segment, the lower arm can be represented by a second rigid body segment, the palm can be represented by a third rigid body segment, and each finger on the hand can be represented by at least one rigid body segment. The set of joint angles between the rigid body segments connected in the musculoskeletal model can define the orientation of each connected rigid body segment relative to each other and a reference system (such as a body trunk). As new sensor data is measured and processed by the inference model to provide a new prediction of the musculoskeletal position information (e.g., a set of updated joint angles), the computer-based musculoskeletal representation of the user's body can be updated based on the set of updated joint angles, which is determined based on the output of the inference model. In this way, the computer-based musculoskeletal representation can be dynamically updated in real time as sensor data is continuously recorded.
[0088] The computer-based musculoskeletal representation may be represented and stored using any suitable device and method. For example, the computer-based musculoskeletal representation may be stored in a memory (e.g., Figure 8 Although referred to herein as a "musculoskeletal" representation to reflect that muscle activity may be associated with the representation, some musculoskeletal representations may correspond to bone structures, muscle structures, or a combination of bone structures and muscle structures in the body.
[0089] In some examples, direct measurements of neuromuscular activity and / or muscle activity that underlie the user's movement can be combined with a generated musculoskeletal representation. Measurements from multiple sensors placed on the user's body can be used to create a unified representation of muscle recruitment by overlaying the measurements onto the skeleton that dynamically forms the posture. In some examples, muscle activity sensed by neuromuscular sensors and / or information derived from muscle activity (e.g., force information) can be combined in real time with a computer-generated musculoskeletal representation.
[0090] Figure 6 A method 600 for determining body state information based on recorded sensor data according to an embodiment of the present disclosure is shown. Figure 6, but the operations of method 600 may be performed in any order. In operation 602, as described above, sensor data may be recorded by one or more sensors and provided as input to one or more trained inference models for predicting a physical state. In some examples, the sensors may include a plurality of neuromuscular sensors (e.g., EMG sensors) arranged on a wearable device worn by a user. For example, the EMG sensors may be arranged (e.g., circumferentially) on an elastic band that is configured to be worn around a wrist or forearm of a user to record neuromuscular signals from the user as the user applies force and / or performs various movements, postures, and / or gestures. Examples of wearable devices that may be used in accordance with embodiments of the present disclosure include Figure 7 Wearable devices 700, Figure 8 Wearable devices 800, Fig.13 Wearable devices 1320, Fig.14 The wearable device 1404 will be described in more detail below.
[0091] Additionally or alternatively, some embodiments may further include one or more auxiliary sensors configured to continuously record auxiliary signals, which may also be provided as input to one or more trained inference models. Examples of auxiliary sensors may include, but are not limited to, IMU sensors, imaging devices, radiation detection devices (e.g., laser scanning devices), heart rate monitors, any other type of biosensor configured to continuously record biophysical information from a user during the execution of one or more movements or gestures, or combinations thereof.
[0092] The method 600 may proceed to operation 604, where, optionally, derived signal data is determined based on the signals recorded by the sensors. For example, accelerometer data recorded by one or more IMU sensors may be integrated and / or filtered to determine derived signal data associated with one or more muscles during performance of a gesture. The derived signal data may also be provided as an input to the trained inference model in addition to or as an alternative to the raw signal data or otherwise processed raw signal data recorded by the sensors.
[0093] Method 600 may proceed to operation 606, where body state information is determined based on the output of the trained reasoning model. The gestures performed by the user may include discrete gestures (such as placing the user's palm down on the table) and / or continuously moving gestures (such as waving fingers back and forth). During the user's movement (including during the execution of the gesture), neuromuscular signals may be continuously recorded and may be continuously provided to the trained reasoning model as input, generating a real-time estimate of the position and / or force of the user's body parts (i.e., body state information) as the output of the trained reasoning model. Method 600 may proceed to operation 608, where the real-time body state prediction output output from the trained reasoning model is used to update the musculoskeletal representation associated with the hand. In some examples, the musculoskeletal representation represents rigid segments in the hand and joints connecting these rigid segments. In other embodiments, the musculoskeletal representation may include at least some rigid segments corresponding to the arm connected to the hand. Accordingly, the phrase "musculoskeletal representation associated with the hand" should be understood as a musculoskeletal representation including a musculoskeletal representation of the hand and / or a musculoskeletal representation including a hand and a representation of at least a portion of the arm connected to the hand.
[0094] Figure 7 A perspective view of an example wearable device 700 is shown, the device 700 includes sixteen sensors 710 (e.g., EMG sensors) arranged circumferentially around an elastic band 720, which is configured to be worn around a user's body part (e.g., the user's lower arm or wrist). As shown, the sensors 710 can be arranged circumferentially around the elastic band 720. Any suitable number of sensors 710 can be used. The number and arrangement of the sensors 710 can depend on the specific application used for the wearable device. For example, a wearable armband or wristband can be used to generate control information to control an artificial reality system, a robot, a vehicle, scroll through text, control a virtual avatar, or for any other suitable control task.
[0095] In some examples, sensor 710 may include a collection of neuromuscular sensors (e.g., EMG sensors). In other embodiments, sensor 710 may include a collection of neuromuscular sensors and at least one "auxiliary" sensor configured to record auxiliary signals (e.g., periodically, continuously, or as needed). Examples of auxiliary sensors may include, but are not limited to, other sensors such as IMU sensors, microphones, imaging sensors (e.g., cameras), radiation-based sensors, laser scanning devices, or other types of sensors, such as heart rate monitors.
[0096] In some examples, hardware signal processing circuitry can be used to process the output of one or more sensing components (e.g., sensor 710) (e.g., to perform amplification, filtering, and / or rectification). In some examples, at least some signal processing of the output of the sensing components can be performed in software. Thus, signal processing of signals sampled by the sensors can be performed in hardware, software, or any suitable combination of hardware and software, as aspects of the techniques described herein are not limited in this regard. Figure 8 Non-limiting examples of signal processing systems for processing data recorded from sensor 710 are discussed in more detail.
[0097] Figure 8 FIG. 8 shows an example block diagram of a wearable system 800 having multiple sensors according to an embodiment of the present disclosure. Figure 8 As shown, the wearable system 800 may include a wearable device 810, a head mounted display (HMD) 826, and an adapter (dongle) 840. The wearable device 810, the HMD 826, and the adapter 840 may communicate with each other via wireless communication (e.g., via Bluetooth TM or other suitable short-range wireless communication technology) or wired communication. Wearable device 810 may include sensor 812 (e.g., EMG sensor), examples of which are described above with reference to Figure 5 and Figure 6 816. The data from the sensor 812 and / or the data from the sensor of the HMD 826 can be used to generate basic fact data (e.g., labeled time series data). The output of the sensor 812 can be provided to the analog front end 814, which can be configured to perform analog signal processing (e.g., noise reduction, filtering, amplification, etc.) on the recorded signal from the sensor 812. The processed analog signal from the analog front end 814 can be provided to the analog-to-digital converter (ADC) 816, which can convert the analog signal into a digital signal so that the signal can be processed by the processor 822 and / or the processor 830 of the HMD 826.
[0098] Processor 822 and / or processor 830 (e.g., a microcontroller, a central processing unit, a digital signal processor, a graphics processor, etc.) may execute instructions stored in memory 821 for implementing the method of the present disclosure (including but not limited to, generating one or more training data sets by time-shifting neuromuscular activity time series data and / or label time series data received from sensor 812 by one or more time intervals, training one or more inference models based on neuromuscular activity time series data using the one or more training data sets, and configuring a state prediction system to use the trained inference model to predict the state of a body part of the user). Figure 8As shown, the processor 822 may also receive input from other sensors (e.g., IMU sensor 818, image sensor, etc.) that may be configured to track the position of a user's body part. The processor 822 and other electronic components of the wearable device 810 may be powered by a battery 820. The output of the signal processing performed by the processor 822 (e.g., a musculoskeletal representation of the user's body) may be provided to a transceiver 824 for transmission to an adapter 840 and / or an HMD 826.
[0099] The adapter 840 may include a transceiver 834 configured to communicate with the transceiver 824 of the wearable device 810 and / or the transceiver 832 of the HMD 826. Communication between the transceivers 834, 824, and 828 may use any suitable wireless technology and protocol, non-limiting examples of which include WiFi, near field communication, and / or Bluetooth. TM .Bluetooth TM The radio 836 may be configured to act as a gateway device to coordinate communications between the various wearable devices of the system 800, including the HMD 826 and the wearable device 810. In additional embodiments, the wearable device 810, the HMD 826, and / or the adapter 840 may communicate with each other via a wired connection.
[0100] Signals received from sensor 812 may be processed using an inference model as described above to predict a body part state of the user's body. HMD 826 may receive the body part state from wearable device 810, and / or instructions executed on a processor 830 of HMD 826 may use one or more trained inference models to determine the body part state. Processor 830 of HMD 826 may use the determined body part state to generate a visual representation of the body part state of the user of wearable device 810. The visual representation of the user's body part state may be displayed to the user on a display 828 of HMD 826. The visual representation of the user's body part state displayed to the user wearing HMD 826 may be combined with an artificial reality application. In some examples, HMD 826 may be Fig.11 Glasses equipment 1102, Fig.12 Virtual reality system 1200, Fig.14 HMD1402.
[0101] Fig. 9is a flow chart illustrating an example method 900 for predicting a physical state based on neuromuscular data. At operation 910, method 900 may include receiving neuromuscular activity data over a first time series from a first sensor on a wearable device worn by a user. Operation 910 may be performed in a variety of ways, for example, a neuromuscular sensor of the wearable device may periodically generate a time series of data indicating the neuromuscular activity of the user.
[0102] At operation 920, method 900 may include receiving basic fact data from a second different sensor, the basic fact data indicating a body part state of a user's body part over a second time series. Operation 920 may be performed in a variety of ways. For example, the basic fact data may be a labeled time series data indicating a body part state of the user while performing a task. The body part state may be or include position information corresponding to the spatial position of different body segments of the user during task performance. The position information may be obtained using one or more external devices (e.g., a camera, an IMU) that tracks the position of different points on the user's body during task performance.
[0103] At operation 930, method 900 may include generating one or more training data sets by time-shifting at least a portion of the neuromuscular activity data on the first time series relative to the second time series to associate the neuromuscular activity data with at least a portion of the basic fact data. Operation 930 may be performed in a variety of ways. For example, appropriate time intervals may be identified by generating multiple training data sets with multiple time offsets. The time offsets may be different corresponding time intervals based on factors including the electromechanical delay time of the user (e.g., the muscle response time of the user) and / or the known characteristic latency of the system. The time-shift interval may determine the system latency and may be based on the accuracy requirements of the task. For example, a task that prioritizes precise movement (e.g., remote surgery) may accept a greater latency in exchange for higher accuracy, while a task that prioritizes fast movement (e.g., video games) may accept a lower accuracy in exchange for a lower latency.
[0104] At operation 940, method 900 may include training one or more inference models based on one or more training data sets. Operation 940 may be performed in a variety of ways. For example, a sequence of data sets may be used as input to train an inference model, and each data set in the sequence may include an n-dimensional vector of sensor data (e.g., sensor data from a neuromuscular sensor, an IMU sensor, etc.). The inference model may provide an output indicating the likelihood that a musculoskeletal representation of the user's body will be characterized by a set of musculoskeletal positional characteristics for each task or movement performed by the user. The inference model may be used to predict body states and create musculoskeletal representations associated with a user's body parts. A visual representation of a user's body parts may be displayed to the user. For example, a visual representation of a user's body parts may be displayed to the user on a head-mounted display.
[0105] Therefore, the present disclosure includes systems, methods, and apparatus that can be used to predict the state of a user's body part. For example, an artificial reality system may include a wearable device that includes a sensor and system configured to predict the state of a user's body part. A virtual representation of the predicted state of a body part (e.g., a hand) can be displayed to the user on an HMD. The HMD can also display a virtual object (e.g., a game controller, a moving object) held by a virtual representation of the hand. Compared to traditional systems, a virtual representation of the predicted state of a body part displayed to the user in conjunction with the audio / video content of the artificial reality application can create a more compelling artificial reality experience (such as by reducing the waiting time between predicted and actual body movements).
[0106] The above embodiments may be implemented in any of a variety of ways. For example, the embodiments may be implemented using hardware, software, or a combination thereof. When implemented in software, the software code may be executed on any suitable processor or processor cluster (whether provided in a single computer or distributed between multiple computers). It should be appreciated that any component or component cluster that performs the above functions may be generally considered to be one or more controllers that control the functions discussed above. One or more controllers may be implemented in a variety of ways (such as using dedicated hardware or using one or more processors that are programmed to perform the functions listed above using microcode or software).
[0107] In this regard, it should be understood that one implementation of an embodiment of the present invention includes at least one non-transitory computer-readable storage medium (e.g., computer memory, portable memory, optical disk, etc.) encoded with a computer program (e.g., multiple instructions), which, when executed on a processor, performs the functions discussed above of an embodiment of the present invention. The computer-readable storage medium may be removable so that the program stored thereon may be loaded onto any computer resource to implement the various aspects of the present invention discussed herein. In addition, it should be recognized that reference to a computer program that, when executed, performs the functions discussed above is not limited to an application program running on a host computer. Instead, the term "computer program" is used in a general sense herein to refer to any type of computer code (e.g., software or microcode) that can be used to program a processor to implement the various aspects of the present invention discussed above.
[0108] The various aspects of the present invention may be used alone, in combination, or in various arrangements not specifically discussed in the previously described embodiments, and therefore their application is not limited to the details and arrangements of the components set forth in the foregoing description or shown in the accompanying drawings. For example, aspects described in one embodiment may be combined in any manner with aspects described in other embodiments.
[0109] In addition, embodiments of the present invention may be implemented as one or more methods, examples of which have been provided. The actions performed as part of the method may be ordered in any suitable manner. Accordingly, embodiments may be constructed in which the actions are performed in a different order than shown, which may include performing some actions simultaneously, even though shown as sequential actions in the illustrative embodiments.
[0110] The use of ordinal terms such as "first," "second," "third," etc. in the claims to modify claim elements does not by itself imply any priority, precedence, or order of one claim element relative to another claim element, or the temporal order in which the actions of the method are performed. Such terms are merely used as labels to distinguish one claim element with a particular name from another element with the same name (but for the use of ordinal terms).
[0111] The phraseology and terminology used herein are for the purpose of description and should not be regarded as limiting. The use of "including," "comprising," "having," "containing," "involving," and variations thereof is meant to include the items listed thereafter and additional items.
[0112] Embodiments of the present disclosure may include or be implemented in conjunction with various types of artificial reality systems. Artificial reality is a form of reality that has been adjusted in some way before being presented to a user, which may include, for example, virtual reality, augmented reality, mixed reality, hybrid reality, or some combination and / or derivative thereof. Artificial reality content may include fully generated content or content generated in combination with captured (e.g., real-world) content. Artificial reality content may include video, audio, tactile feedback, or some combination thereof, any of which may be presented in a single channel or multiple channels (such as stereoscopic video that produces a three-dimensional (3D) effect to the audience). In addition, in some embodiments, artificial reality may also be associated with applications, products, accessories, services, or some combination thereof, which are used, for example, to create content in artificial reality and / or to be used in other ways in artificial reality (e.g., to perform activities in artificial reality).
[0113] Artificial reality systems may be implemented in a variety of different forms and configurations. Some artificial reality systems may be designed to operate without a near-eye display (NED). Other artificial reality systems may include a NED that also provides visibility into the real world (e.g., Fig.10 ), or an NED that visually immerses a user in an artificial reality (e.g., Fig.11 100). While some artificial reality devices may be stand-alone systems, other artificial reality devices may communicate and / or coordinate with external devices to provide an artificial reality experience to a user. Examples of such external devices include handheld controllers, mobile devices, desktop computers, devices worn by a user, devices worn by one or more other users, and / or any other suitable external system.
[0114] The embodiments discussed in this disclosure may also be implemented in an augmented reality system including one or more NEDs. Fig.10 As shown, the augmented reality system 1000 may include an eyewear device 1002 having a frame 1010, and the frame 1010 is configured to hold a left display device 1015 (A) and a right display device 1015 (B) in front of the user's eyes. The display devices 1015 (A) and 1015 (B) can function together or independently to present an image or a series of images to the user. Although the augmented reality system 1000 includes two displays, embodiments of the present disclosure can be implemented in an augmented reality system having a single NED or more than two NEDs.
[0115] In some embodiments, the augmented reality system 1000 may include one or more sensors, such as sensor 1040. Sensor 1040 may generate measurement signals in response to the movement of the augmented reality system 1000 and may be located on substantially any portion of the frame 1010. Sensor 1040 may represent a position sensor, an inertial measurement unit (IMU), a depth camera assembly, or any combination thereof. In some embodiments, the augmented reality system 1000 may include or exclude sensor 1040, or may include more than one sensor. In embodiments where sensor 1040 includes an IMU, the IMU may generate calibration data based on the measurement signals from sensor 1040. Examples of sensor 1040 may include, but are not limited to, accelerometers, gyroscopes, magnetometers, other suitable types of sensors that detect movement, sensors for error correction of an IMU, or some combination thereof.
[0116] The augmented reality system 1000 may also include a microphone array having a plurality of acoustic transducers 1020(A)-1020(J) (collectively referred to as acoustic transducers 1020). The acoustic transducers 1020 may be transducers that detect changes in air pressure caused by sound waves. Each acoustic transducer 1020 may be configured to detect sound and convert the detected sound into an electronic format (e.g., an analog or digital format). Fig.10 The microphone array in can include, for example, ten acoustic transducers: 1020(A) and 1020(B), which can be designed to be placed in the user's respective ears; acoustic transducers 1020(C), 1020(D), 1020(E), 1020(F), 1020(G) and 1020(H), which can be located at different positions on the frame 1010; and / or acoustic transducers 1020(I) and 1020(J), which can be located on the respective neckbands 1005.
[0117] In some embodiments, one or more of the acoustic transducers 1020(A)-1020(F) can be used as output transducers (e.g., speakers). For example, the acoustic transducers 1020(A) and / or 1020(B) can be earbuds or any other suitable type of earphones or speakers.
[0118] The configuration of the acoustic transducers 1020 of the microphone array may vary. Fig.101020, but the number of acoustic transducers 1020 may be greater or less than ten. In some embodiments, using a greater number of acoustic transducers 1020 may increase the amount of audio information collected and / or the sensitivity and accuracy of the audio information. Conversely, using a smaller number of acoustic transducers 1020 may reduce the computing power required for the associated controller 1050 to process the collected audio information. In addition, the position of each acoustic transducer 1020 of the microphone array may vary. For example, the position of the acoustic transducer 1020 may include a defined position on the user, a defined coordinate on the frame 1010, an orientation associated with each acoustic transducer 1020, or some combination thereof.
[0119] The acoustic transducers 1020(A) and 1020(B) may be located in different parts of the user's ear, such as behind the auricle or in the pinna or fossa. Alternatively, in addition to the acoustic transducer 1020 in the ear canal, there may be additional acoustic transducers 1020 on or around the ear. Positioning the acoustic transducer 1020 near the user's ear canal enables the microphone array to collect information about how the sound reaches the ear canal. By positioning at least two acoustic transducers 1020 on either side of the user's head (e.g., as binaural microphones), the augmented reality device 1000 can simulate binaural hearing and capture a 3D stereo sound field around the user's head. In some embodiments, the acoustic transducers 1020(A) and 1020(B) may be connected to the augmented reality system 1000 via a wired connection 1030, and in other embodiments, the acoustic transducers 1020(A) and 1020(B) may be connected to the augmented reality system 1000 via a wireless connection (e.g., a Bluetooth connection). In other embodiments, acoustic transducers 1020 (A) and 1020 (B) may not be used in conjunction with augmented reality system 1000 at all.
[0120] The acoustic transducers 1020 on the frame 1010 may be positioned along the length of the temples, across the bridge, above or below the display devices 1015(A) and 1015(B), or some combination thereof. The acoustic transducers 1020 may be oriented so that the microphone array can detect sounds in a wide range of directions around the user wearing the augmented reality system 1000. In some embodiments, an optimization process may be performed during the manufacture of the augmented reality system 1000 to determine the relative positioning of each acoustic transducer 1020 in the microphone array.
[0121] In some examples, the augmented reality system 1000 can include or be connected to an external device (e.g., a paired device), such as a neckband 1005. The neckband 1005 generally represents any type or form of paired device. Therefore, the following discussion of the neckband 1005 is also applicable to various other paired devices, such as charging boxes, smart watches, smart phones, wristbands, other wearable devices, handheld controllers, tablet computers, laptop computers, and other external computing devices.
[0122] As shown, the neckband 1005 can be connected to the eyeglass device 1002 via one or more connectors. The connector can be wired or wireless and can include electrical and / or non-electrical (e.g., structural) components. In some cases, the eyeglass device 1002 and the neckband 1005 can operate independently without any wired or wireless connection between them. Although Fig.10 Components of the eyeglass device 1002 and neckband 1005 are shown in example locations on the eyeglass device 1002 and neckband 1005, but these components may be located elsewhere on the eyeglass device 1002 and / or neckband 1005 and / or distributed differently on the eyeglass device 1002 and / or neckband 1005. In some embodiments, components of the eyeglass device 1002 and neckband 1005 may be located on one or more additional peripheral devices that are paired with the eyeglass device 1002, neckband 1005, or some combination thereof.
[0123] Pairing an external device (such as a neckband 1005) with an augmented reality eyewear device can enable the eyewear device to achieve the form factor of a pair of glasses while still providing sufficient battery and computing power to expand functionality. Some or all of the battery power, computing resources, and / or additional features of the augmented reality system 1000 can be provided by a paired device, or shared between a paired device and the eyewear device, thereby reducing the weight, thermal distribution, and form factor of the eyewear device as a whole while still maintaining the desired functionality. For example, the neckband 1005 can allow components that would otherwise be included on the eyewear device to be included in the neckband 1005 because the user can tolerate a heavier weight load on their shoulders than they would tolerate on their heads. The neckband 1005 can also have a larger surface area to diffuse and disperse heat into the surrounding environment over that surface area. Therefore, the neckband 1005 can allow for a larger battery and computing capacity than would otherwise be available on a standalone eyewear device. Because the weight carried in the neckband 1005 is less harmful to the user than the weight carried in the eyeglass device 1002, the user can tolerate wearing a lighter eyeglass device and can tolerate carrying or wearing a paired device for longer periods of time than wearing a heavier stand-alone eyeglass device, thereby enabling the user to more fully integrate the artificial reality environment into their daily activities.
[0124] The neckband 1005 can be communicatively coupled to the eyewear device 1002 and / or other devices. These other devices can provide certain functions (e.g., tracking, positioning, depth mapping, processing, storage, etc.) to the augmented reality system 1000. Fig.10 In an embodiment, the neckband 1005 may include two acoustic transducers (e.g., 1020 (I) and 1020 (J)) that are part of a microphone array (or potentially form their own microphone subarray). The neckband 1005 may also include a controller 1025 and a power supply 1035.
[0125] The acoustic transducers 1020(I) and 1020(J) of the neckband 1005 may be configured to detect sound and convert the detected sound into an electronic format (analog or digital). Fig.10 In an embodiment of the present invention, the acoustic transducers 1020(I) and 1020(J) can be positioned on the neckband 1005, thereby increasing the distance between the neckband acoustic transducers 1020(I) and 1020(J) and other acoustic transducers 1020 positioned on the eyeglass device 1002. In some cases, increasing the distance between the acoustic transducers 1020 of the microphone array can improve the accuracy of the beamforming performed via the microphone array. For example, if the acoustic transducers 1020(C) and 1020(D) detect a sound, and the distance between the acoustic transducers 1020(C) and 1020(D) is greater than, for example, the distance between the acoustic transducers 1020(D) and 1020(E), the determined source location of the detected sound may be more accurate than if the acoustic transducers 1020(D) and 1020(E) detect the sound.
[0126] The controller 1025 of the neckband 1005 can process information generated by sensors on the neckband 1005 and / or the augmented reality system 1000. For example, the controller 1025 can process information from the microphone array describing the sound detected by the microphone array. For each detected sound, the controller 1025 can perform a direction of arrival (DOA) estimation to estimate the direction from which the detected sound arrives at the microphone array. When the microphone array detects a sound, the controller 1025 can fill the audio data set with this information. In an embodiment where the augmented reality system 1000 includes an inertial measurement unit, the controller 1025 can calculate all inertial and spatial calculations from the IMU located on the eyeglass device 1002. The connector can transfer information between the augmented reality system 1000 and the neckband 1005 and between the augmented reality system 1000 and the controller 1025. The information can be in the form of optical data, electrical data, wireless data, or any other transmittable data form. Moving the processing of the information generated by the augmented reality system 1000 to the neckband 1005 can reduce the weight and heat in the eyeglass device 1002, making it more comfortable for the user.
[0127] The power supply 1035 in the neckband 1005 can provide power to the eyeglass device 1002 and / or the neckband 1005. The power supply 1035 can include, but is not limited to, a lithium-ion battery, a lithium polymer battery, a primary lithium battery, an alkaline battery, or any other form of power storage device. In some cases, the power supply 1035 can be a wired power supply. Including the power supply 1035 on the neckband 1005 instead of on the eyeglass device 1002 can help better distribute the weight and heat generated by the power supply 1035.
[0128] As mentioned above, some artificial reality systems can replace one or more of the user's sensory perceptions of the real world with a virtual experience, rather than mixing the artificial reality with the actual reality. An example of this type of system is a head-mounted display system, such as Fig.11 The virtual reality system 1100 in FIG. 1 mainly or completely covers the user's field of view. The virtual reality system 1100 may include a front rigid body 1102 and a band 1104 shaped to surround the user's head. The virtual reality system 1100 may also include output audio transducers 1106 (A) and 1106 (B). In addition, although in Fig.11 Not shown, but front rigid body 1102 may include one or more electronic components, including one or more electronic displays, one or more inertial measurement units (IMUs), one or more tracking transmitters or detectors, and / or any other suitable device or system for creating an artificial reality experience.
[0129] Artificial reality systems may include various types of visual feedback mechanisms. For example, the display device in the augmented reality system 1000 and / or the virtual reality system 1100 may include one or more liquid crystal displays (LCDs), light emitting diode (LED) displays, organic LED (OLED) displays, and / or any other suitable type of display screen. The artificial reality system may include a single display screen for both eyes, or a display screen may be provided for each eye, which may provide additional flexibility for zoom adjustment or for correcting a user's refractive error. Some artificial reality systems may also include an optical subsystem having one or more lenses (e.g., a conventional concave or convex lens, a Fresnel lens, an adjustable liquid lens, etc.) through which a user can view a display screen.
[0130] In addition to or instead of using a display screen, some artificial reality systems may include one or more projection systems. For example, a display device in the augmented reality system 1000 and / or the virtual reality system 1100 may include a micro-LED projector that projects light (using, for example, a waveguide) into the display device, such as a transparent combination lens that allows ambient light to pass through. The display device can refract the projected light toward the user's pupil, and can enable the user to view artificial reality content and the real world simultaneously. The artificial reality system can also be configured with any other suitable type or form of image projection system.
[0131] The artificial reality system may also include various types of computer vision components and subsystems. For example, the augmented reality system 1000 and / or the virtual reality system 1100 may include one or more optical sensors, such as a two-dimensional (2D) or 3D camera, a time-of-flight depth sensor, a single beam or scanning laser rangefinder, a 3D LiDAR sensor, and / or any other suitable type or form of optical sensor. The artificial reality system may process data from one or more of these sensors to identify the user's location, map the real world, provide the user with context about the real-world environment, and / or perform various other functions.
[0132] The artificial reality system may also include one or more input and / or output audio transducers. Fig.11 In the example shown, the output audio transducers 1106(A) and 1106(B) may include voice coil speakers, ribbon speakers, electrostatic speakers, piezoelectric speakers, bone conduction transducers, cartilage conduction transducers, and / or any other suitable type or form of audio transducers. Similarly, the input audio transducers may include condenser microphones, dynamic microphones, ribbon microphones, and / or any other type or form of input transducers. In some embodiments, a single transducer may be used for both audio input and audio output.
[0133] Although in Fig.11 Not shown, but the artificial reality system may include a tactile (i.e., haptic) feedback system that can be incorporated into headgear, gloves, tights, handheld controllers, environmental equipment (e.g., chairs, floor mats, etc.), and / or any other type of device or system. The tactile feedback system can provide various types of skin feedback, including vibration, force, traction, texture, and / or temperature. The tactile feedback system can also provide various types of kinesthetic feedback, such as motion and compliance. Tactile feedback can be implemented using motors, piezoelectric actuators, fluidic systems, and / or various other types of feedback mechanisms. The tactile feedback system can be implemented independently of other artificial reality devices, within other artificial reality devices, and / or in combination with other artificial reality devices.
[0134] By providing tactile sensations, auditory content, and / or visual content, artificial reality systems can create complete virtual experiences or enhance the user's real-world experience in various contexts and environments. For example, an artificial reality system can help or expand a user's perception, memory, or cognition within a specific environment. Some systems can enhance the user's interaction with other people in the real world, or can enable a more immersive interaction between the user and other people in the virtual world. Artificial reality systems can also be used for educational purposes (e.g., for teaching or training in schools, hospitals, government organizations, military organizations, commercial enterprises, etc.), entertainment purposes (e.g., for playing video games, listening to music, watching video content, etc.), and / or for accessibility purposes (e.g., as hearing aids, visual aids, etc.). The embodiments disclosed herein can implement or enhance the user's artificial reality experience in one or more of these contexts and environments and / or in other contexts and environments.
[0135] As described above, artificial reality systems 1000 and 1100 can be used with various other types of devices to provide a more compelling artificial reality experience. These devices can be tactile interfaces with sensors that provide tactile feedback and / or collect tactile information about the user's interaction with the environment. The artificial reality systems disclosed herein may include various types of tactile interfaces that detect or convey various types of tactile information, including tactile feedback (e.g., feedback detected by the user through nerves in the skin, also referred to as skin feedback) and / or kinesthetic feedback (e.g., feedback detected by the user through receptors located in muscles, joints, and / or tendons).
[0136] Haptic feedback can be provided by interfaces located within the user's environment (e.g., a chair, table, floor, etc.) and / or interfaces on items that the user can wear or carry (e.g., gloves, wristbands, etc.). As an example, Fig.12 A vibrotactile system 1200 in the form of a wearable glove (haptic device 1210) and a wristband (haptic device 1220) is shown. The haptic device 1210 and the haptic device 1220 are shown as examples of wearable devices, which include a flexible, wearable textile material 1230 that is shaped and configured to be positioned against a user's hand and wrist, respectively. The present disclosure also includes a vibrotactile system that can be shaped and configured to be positioned against other human body parts, such as fingers, arms, heads, torsos, feet, or legs. As an example and not limitation, the vibrotactile system according to various embodiments of the present disclosure may also be in the form of gloves, headbands, armbands, sleeves, head covers, socks, shirts, or pants, and there are other possibilities. In some examples, the term "textile" may include any flexible, wearable material, including woven fabrics, non-woven fabrics, leather, cloth, flexible polymer materials, composite materials, and the like.
[0137] One or more vibrotactile devices 1240 may be at least partially located within one or more corresponding pockets formed in textile material 1230 of vibrotactile system 1200. Vibrotactile devices 1240 may be positioned to provide a vibrotactile sensation (e.g., tactile feedback) to a user of vibrotactile system 1200. For example, vibrotactile devices 1240 may be positioned to rest against a user's finger, thumb, or wrist, such as Fig.12 In some examples, vibrotactile device 1240 can be flexible enough to conform or bend with a corresponding body part of the user.
[0138] A power source 1250 (e.g., a battery) for applying a voltage to vibrating tactile device 1240 to activate vibrating tactile device 1240 can be electrically coupled to vibrating tactile device 1240, such as via conductive line 1252. In some examples, each vibrating tactile device 1240 can be independently electrically coupled to power source 1250 for individual activation. In some embodiments, processor 1260 can be operably coupled to power source 1250 and configured (e.g., programmed) to control the activation of vibrating tactile device 1240.
[0139] The vibration tactile system 1200 can be implemented in a variety of ways. In some examples, the vibration tactile system 1200 can be an independent system with integrated subsystems and components that run independently of other devices and systems. As another example, the vibration tactile system 1200 can be configured to interact with another device or system 1270. For example, in some examples, the vibration tactile system 1200 may include a communication interface 1280 for receiving signals and / or sending signals to other devices or systems 1270. Other devices or systems 1270 may be mobile devices, game consoles, artificial reality (e.g., virtual reality, augmented reality, mixed reality) devices, personal computers, tablet computers, network devices (e.g., modems, routers, etc.), handheld controllers, etc. The communication interface 1280 can realize the communication between the vibration tactile system 1200 and other devices or systems 1270 via wireless (e.g., Wi-Fi, Bluetooth, cellular, radio, etc.) links or wired links. If present, communication interface 1280 may communicate with processor 1260 , such as providing signals to processor 1260 to activate or deactivate one or more vibrotactile devices 1240 .
[0140] The vibrotactile system 1200 may optionally include other subsystems and components, such as a touch-sensitive pad 1290, a pressure sensor, a motion sensor, a position sensor, an illumination element, and / or a user interface element (e.g., an on / off button, a vibration control element, etc.). During use, the vibrotactile device 1240 may be configured to be activated for a variety of different reasons, such as in response to a user's interaction with a user interface element, a signal from a motion or position sensor, a signal from the touch-sensitive pad 1290, a signal from a pressure sensor, a signal from other devices or systems 1270, etc.
[0141] Although the power supply 1250, the processor 1260 and the communication interface 1280 are Fig.12 12 is shown as being located in the haptic device 1220, but the present disclosure is not limited thereto. For example, one or more of the power supply 1250, the processor 1260, or the communication interface 1280 may be located within the haptic device 1210 or within another wearable textile.
[0142] Tactile wearable devices, such as Fig.12 What is shown and described may be implemented in various types of artificial reality systems and environments. Fig.13 An exemplary artificial reality environment 1300 including one head-mounted virtual reality display and two haptic devices (i.e., gloves) is shown, and in other embodiments, any number and / or combination of these and other components may be included in the artificial reality system. For example, in some embodiments, there may be multiple head-mounted displays, each with an associated haptic device, each head-mounted display and each haptic device communicating with the same console, portable computing device, or other computing system.
[0143] Head mounted display 1302 generally represents any type or form of virtual reality system, such as Fig.111300 in a virtual reality system 1100. The haptic device 1304 generally represents any type or form of wearable device worn by a user of the artificial reality system that provides tactile feedback to the user to give the user the feeling that he or she is physically in contact with a virtual object. In some embodiments, the haptic device 1304 can provide tactile feedback by applying vibration, motion, and / or force to the user. For example, the haptic device 1304 can limit or enhance the movement of the user. To give a specific example, the haptic device 1304 can limit the forward movement of the user's hand so that the user has the feeling that his or her hand has been in physical contact with a virtual wall. In this particular example, one or more actuators within the haptic device can achieve physical movement restriction by pumping fluid into an inflatable bladder of the haptic device. In some examples, the user can also use the haptic device 1304 to send an action request to the console. Examples of action requests include, but are not limited to, requests to start an application and / or end an application and / or requests to perform a specific action within an application.
[0144] Although tactile interfaces can be used in virtual reality systems, such as Fig.13 As shown, tactile interfaces can also be used in augmented reality systems, such as Fig.14 shown. Fig.14 14 is a perspective view of a user 1410 interacting with an augmented reality system 1400. In this example, the user 1410 may wear a pair of augmented reality glasses 1420, which may have one or more displays 1422 and be paired with a haptic device 1430. The haptic device 1430 may be a wristband that includes a plurality of strap elements 1432 and a tensioning mechanism 1434 that connects the strap elements 1432 to each other.
[0145] One or more band elements 1432 may include any type or form of actuator suitable for providing tactile feedback. For example, one or more band elements 1432 may be configured to provide one or more different types of skin feedback, including vibration, force, traction, texture and / or temperature. In order to provide such feedback, band elements 1432 may include one or more actuators of various types. In one example, each band element 1432 may include a vibrating tactile sensor that is configured to vibrate in unison or independently to provide one or more various types of tactile sensations to the user. Optionally, only a single band element or a subset of band elements may include a vibrating tactile sensor.
[0146] The haptic devices 1210, 1220, 1304, and 1430 may include any suitable number and / or type of tactile transducers, sensors, and / or feedback mechanisms. For example, the haptic devices 1210, 1220, 1304, and 1430 may include one or more mechanical transducers, piezoelectric transducers, and / or fluid transducers. The haptic devices 1210, 1220, 1304, and 1430 may also include various combinations of transducers of different types and forms that work together or independently to enhance the user's artificial reality experience. In one example, each band element 1432 of the haptic device 1430 may include a vibrating tactile (e.g., a vibrating tactile actuator) that is configured to vibrate in unison or independently to provide the user with one or more various types of tactile sensations.
[0147] As non-limiting examples, the following embodiments are included in the present disclosure.
[0148] Example 1: A method comprising: receiving neuromuscular activity data on a first time series from a first sensor on a wearable device worn by a user, receiving basic fact data on a second time series from a second sensor, the basic fact data indicating a body part state of a body part of the user, generating one or more training data sets by time-shifting at least a portion of the neuromuscular activity data on the first time series relative to the second time series to associate the neuromuscular activity data with at least a portion of the basic fact data, and training one or more inference models based on the one or more training data sets.
[0149] Example 2: The method of Example 1, further comprising time-shifting the portion of the neuromuscular activity data by one or more time intervals based on electromechanical delays associated with a particular body part of the user.
[0150] Example 3: A method according to Example 1 or 2, wherein the one or more inference models include multiple inference models, and the method further includes: determining a prediction accuracy of each of the multiple inference models, selecting a first inference model from the multiple inference models based on the prediction accuracy determined for each of the multiple inference models, and using the first inference model to predict a body part state of the user.
[0151] Example 4: The method according to any one of Examples 1 to 3, wherein the first inference model is selected further based on a characteristic waiting time associated with a specific body part of the user.
[0152] Example 5: A method according to any one of Examples 1 to 4, wherein determining the prediction accuracy of each of the multiple inference models includes determining the likelihood of correctly estimating the state of the body part using each of the multiple inference models.
[0153] Example 6: A method according to any one of Examples 1 to 5, wherein determining the prediction accuracy of each of multiple inference models includes determining an error value between a known characteristic waiting time associated with a body part state of the user and a waiting time associated with the body part state of the user predicted using each of the multiple inference models.
[0154] Example 7: The method according to any one of Examples 1 to 6 further includes predicting a body part state of the user based on one or more inference models.
[0155] Example 8: The method according to any one of Examples 1 to 7 also includes determining that the predicted body part state of the user corresponds to a specific posture, and performing an action in the artificial reality environment in response to determining that the predicted body part state of the user corresponds to the specific posture.
[0156] Example 9: The method according to any one of Examples 1 to 8 further includes: determining the prediction accuracy of two inference models among one or more inference models, wherein the two inference models correspond to two different time intervals, determining that the prediction accuracy of the two inference models meets a threshold, selecting one of the two inference models corresponding to the larger time interval of the two different time intervals, and using the selected one of the two inference models to predict the state of a body part of the user.
[0157] Example 10: A method according to any one of Examples 1 to 9, wherein the threshold is at least one of: an absolute difference in predictive accuracy of two inference models, or a relative difference in predictive accuracy between a more accurate one of the two inference models and a less accurate one of the two inference models.
[0158] Example 11: The method of any one of Examples 1 to 10, wherein selecting one of the two inference models is also based on a task-related accuracy level.
[0159] Example 12: A method according to any one of Examples 1 to 11, wherein the second sensor is part of a head-mounted display and the second sensor includes at least one of an optical sensor, an inertial measurement sensor, a mutual magnetic induction measurement sensor, or a pressure sensor.
[0160] Example 13: A method according to any one of Examples 1 to 12, wherein the state of a body part of the user is represented as a binary mark indicating the presence or absence of at least one of a posture or a pose.
[0161] Example 14: A method according to any one of Examples 1 to 13, wherein the body part state includes at least one of: a force associated with the body part, a movement of the body part, a posture associated with the body part, a posture associated with the body part, or a posture associated with a part of the body part.
[0162] Example 15: A wearable device comprising: one or more neuromuscular sensors, the one or more neuromuscular sensors configured to record multiple neuromuscular signals from a user wearing the wearable device; and one or more processors programmed to receive neuromuscular activity data over a time series from the one or more neuromuscular sensors, wherein the neuromuscular activity data corresponds to the neuromuscular signals recorded by the one or more neuromuscular sensors, receive position data indicating a state of a body part of the user over the time series, generate one or more training data sets by time-shifting at least one of the neuromuscular activity data or the position data by one or more time intervals, train one or more inference models based at least on the one or more training data sets, and predict the state of the body part of the user based on the trained one or more inference models.
[0163] Example 16: The wearable device of Example 15, wherein the one or more processors are further programmed to select the one or more time intervals based on an electromechanical delay associated with the user's body part.
[0164] Example 17: A wearable device according to Example 15 or Example 16, wherein the one or more inference models include multiple inference models, and the one or more processors are also programmed to determine the prediction accuracy of each of the multiple inference models, select a first inference model from the multiple inference models based on the prediction accuracy determined for each of the multiple inference models, and use the first inference model to predict the state of a body part of the user.
[0165] Example 18: A wearable device according to any one of Examples 15 to 17, wherein one of the multiple inference models is also selected based on a characteristic waiting time of a body part of the user.
[0166] Example 19: A wearable device according to any of Examples 15 to 18, wherein determining the prediction accuracy of each of the multiple inference models includes determining the likelihood of correctly estimating a known body part state using each of the multiple inference models.
[0167] Example 20: An artificial reality system comprising a head-mounted display and a wearable device, the wearable device comprising: one or more neuromuscular sensors, the one or more neuromuscular sensors configured to record multiple neuromuscular signals from a user wearing the wearable device; and one or more processors, the one or more processors programmed to: receive neuromuscular activity data over a time series from the neuromuscular sensors configured to record multiple neuromuscular signals, receive position data indicating a body part state of a body part of the user over the time series, generate one or more training data sets by time-shifting at least one of the neuromuscular activity data or the position data by one or more time intervals, use the one or more training data sets to train one or more inference models based at least on the neuromuscular activity time series data, and use the trained one or more inference models to predict the body part state of the user, wherein the head-mounted display is configured to display a visual representation of the user's body part.
[0168] The foregoing description has been provided to enable others skilled in the art to best utilize the various aspects of the example embodiments disclosed herein. This example description is not intended to be exhaustive or limited to any precise form disclosed. Many modifications and variations are possible without departing from the spirit and scope of the present disclosure. The embodiments disclosed herein should be considered in all respects to be illustrative and not restrictive. In determining the scope of the present disclosure, reference should be made to the attached claims and their equivalents.
[0169] Unless otherwise indicated, the terms "connected to" and "coupled to" (and their derivatives) used in the specification and claims should be interpreted as allowing both direct and indirect (i.e., through other elements or components) connections. In addition, the terms "a" or "an" used in the specification and claims should be interpreted as meaning "at least one of..." Finally, for ease of use, the terms "including" and "having" (and their derivatives) used in the specification and claims are interchangeable with the word "comprising" and have the same meaning.
Claims
1. A method for predicting a body state based on neuromuscular data, the method comprising: receiving neuromuscular activity data over a first time series from a first sensor on a wearable device worn by a user; receiving, from a second sensor, ground truth data over a second time series, the ground truth data indicating a body part state of a body part of a user; generating more than one training data set by time-shifting at least a portion of the neuromuscular activity data on the first time series relative to the second time series by a time interval to associate the neuromuscular activity data with at least a portion of the ground truth data; Based on the more than one training data sets, training more than one inference model; determining a prediction accuracy of two of the more than one inference models, wherein the two inference models are based on two training data sets corresponding to two different time intervals; determining that the prediction accuracy of the two inference models satisfies a threshold; selecting one of the two inference models corresponding to a larger time interval of the two different time intervals; and The state of the body part of the user is predicted using a selected one of the two inference models.
2. The method of claim 1, further comprising time-shifting the portion of the neuromuscular activity data by one or more time intervals based on electromechanical delays associated with a particular body part of the user.
3. The method according to claim 1, wherein: The more than one reasoning models include a plurality of reasoning models; and The method further comprises: determining a predictive accuracy of each of the plurality of inference models; selecting a first inference model from the plurality of inference models based on a prediction accuracy determined for each of the plurality of inference models; and The state of the body part of the user is predicted using the first inference model.
4. The method according to claim 3, wherein: The first inference model is also selected based on a characteristic latency associated with a particular body part of the user.
5. The method according to claim 3, wherein: Determining the predictive accuracy of each of the plurality of inference models includes determining a likelihood of correctly estimating the state of the body part using each of the plurality of inference models.
6. The method according to claim 3, wherein: Determining the prediction accuracy of each of the multiple inference models includes determining an error value between a known characteristic waiting time associated with the body part state of the user and a waiting time associated with the body part state of the user predicted using each of the multiple inference models.
7. The method according to claim 1 also includes predicting the body part state of the user based on the more than one inference models.
8. The method according to claim 7, further comprising: determining that a predicted body part state of the user corresponds to a particular posture; as well as In response to determining that the predicted body part state of the user corresponds to the particular posture, an action is performed in the artificial reality environment.
9. The method according to claim 1, wherein: The threshold is at least one of: an absolute difference in predictive accuracy of the two inference models, or a relative difference in predictive accuracy between a more accurate one of the two inference models and a less accurate one of the two inference models.
10. The method according to claim 1, wherein: The selection of one of the two reasoning models is also based on a task-related accuracy level.
11. The method according to claim 1, wherein: The second sensor is part of a head mounted display, and The second sensor includes at least one of an optical sensor, an inertial measurement sensor, a mutual magnetic induction measurement sensor, or a pressure sensor.
12. The method according to claim 1, wherein: The body part state of the user is represented as a binary flag indicating the presence or absence of at least one of a posture or a pose.
13. The method according to claim 1, wherein: The body part state includes at least one of: a force associated with the body part, a movement of the body part, a posture associated with the body part, a pose associated with the body part, or a pose associated with a portion of the body part.
14. A wearable device comprising: one or more neuromuscular sensors configured to record neuromuscular signals from a user wearing the wearable device; as well as One or more processors programmed to: receiving neuromuscular activity data over a time series from the one or more neuromuscular sensors, wherein the neuromuscular activity data corresponds to the neuromuscular signals recorded by the one or more neuromuscular sensors; receiving position data indicating a state of a body part of a user over the time series; generating more than one training data set by time-shifting at least one of the neuromuscular activity data or the position data by a time interval, the time interval corresponding to the training data set; training more than one inference model based at least in part on the more than one training data sets; determining a prediction accuracy of two of the more than one inference models, wherein the two inference models are based on two training data sets corresponding to two different time intervals; determining that the prediction accuracy of the two inference models satisfies a threshold; selecting one of the two inference models corresponding to a larger time interval of the two different time intervals; and The state of the body part of the user is predicted using a selected one of the two inference models.
15. The wearable device according to claim 14, wherein: The one or more processors are further programmed to select the time interval based on an electromechanical delay associated with a body part of the user.
16. The wearable device of claim 14, wherein: The more than one reasoning models include a plurality of reasoning models; and The one or more processors are also programmed to: determining a predictive accuracy of each of the plurality of inference models; selecting a first inference model from the plurality of inference models based on a prediction accuracy determined for each of the plurality of inference models; as well as The state of the body part of the user is predicted using the first inference model.
17. The wearable device according to claim 16, wherein: One of the plurality of inference models is also selected based on a characteristic latency of a body part of the user.
18. The wearable device according to claim 16, wherein: Determining the predictive accuracy of each of the plurality of inference models includes determining a likelihood of correctly estimating a known body part state using each of the plurality of inference models.
19. An artificial reality system comprising: Head-mounted displays; as well as Wearable devices, including: one or more neuromuscular sensors configured to record a plurality of neuromuscular signals from a user wearing the wearable device; and One or more processors programmed to: receiving neuromuscular activity data over a time series from a neuromuscular sensor configured to record the plurality of neuromuscular signals; receiving position data indicating a body part state of a body part of the user over the time series; generating more than one training data set by time-shifting at least one of the neuromuscular activity data or the position data by a time interval, the time interval corresponding to the training data set; Based at least in part on the more than one training data sets, training more than one inference model; and determining a prediction accuracy of two of the more than one inference models, wherein the two inference models are based on two training data sets corresponding to two different time intervals; determining that the prediction accuracy of the two inference models satisfies a threshold; selecting one of the two inference models corresponding to a larger time interval of the two different time intervals; and The state of the body part of the user is predicted using a selected one of the two inference models, wherein: The head mounted display is configured to display a visual representation of a body part of the user.
