Providing enhanced interaction with physical object using neuromuscular signals in augmented reality environment

Abstract

The invention relates to using neuromuscular signals to provide enhanced interaction with a physical object in an augmented reality environment. Computerized systems, methods, kits, and computer readable media storing code for implementing these methods are provided for interacting with physical objects in an augmented reality (AR) system generated AR environment. One such system includes a plurality of neuromuscular sensors capable of sensing a plurality of neuromuscular signals from a user, and at least one computer processor. Neuromuscular sensors are disposed on one or more wearable devices worn by a user to sense neuromuscular signals. The at least one computer processor is programmed to: determine, based at least in part on a neuromuscular signal sensed by the neuromuscular sensor, information about an interaction of a user with a physical object in an AR environment generated by the AR system; and instructing the AR system to provide feedback based at least in part on the information about the user's interaction with the physical object.

Description

[0001] This application is a divisional application of an application with an application date of October 4, 2019, application number 201980065646.9, and invention name “Using neuromuscular signals to provide enhanced interaction with physical objects in an augmented reality environment”. CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims the benefit under 35 USC § 119(e) of U.S. Provisional Patent Application Serial No. 62 / 741,781, filed on October 5, 2018, entitled “USE OF NEUROMUSCULAR SIGNALS TO PROVIDE ENHANCED INTERACTIONS WITH PHYSICAL OBJECTS IN AN AUGMENTED REALITY ENVIRONMENT,” the entire contents of which are incorporated herein by reference. Field of the Invention

[0003] The present technology relates to systems and methods for detecting and interpreting neuromuscular signals for performing actions in augmented reality (AR) environments and other types of extended reality (XR) environments such as virtual reality (VR) environments, mixed reality (MR) environments, and the like. background

[0004] AR systems provide users with an interactive experience of a real-world environment supplemented by virtual information by superimposing computer-generated perception or virtual information on various aspects of the real-world environment. There are various technologies to control the operation of an AR system. Typically, one or more input devices (such as a controller, keyboard, mouse, camera, microphone, etc.) can be used to control the operation of the AR system. Physical objects in the real-world environment can be annotated with visual indicators in the AR environment generated by the AR system. Visual indicators can provide information about physical objects to users of the AR system. SUMMARY OF THE INVENTION

[0005] According to aspects of the technology described herein, a computerized system for interacting with a physical object in an augmented reality (AR) environment generated by an AR system is provided. The computerized system may include: a plurality of neuromuscular sensors configured to sense a plurality of neuromuscular signals from a user; and at least one computer processor. The plurality of neuromuscular sensors may be arranged on one or more wearable devices worn by the user to sense a plurality of neuromuscular signals. The at least one computer processor may be programmed to: determine information about the user's interaction with a physical object in an AR environment generated by the AR system based at least in part on a plurality of neuromuscular signals sensed by the plurality of neuromuscular sensors; and instruct the AR system to provide feedback based at least in part on the information about the user's interaction with the physical object.

[0006] In one aspect, the at least one computer processor may be further programmed to instruct the AR system to display at least one visual indicator on a physical object in the AR environment.

[0007] In variations of this aspect, at least one computer processor may determine information regarding the user's interaction with the physical object based at least in part on the user's interaction with at least one visual indicator displayed on the physical object.

[0008] In one aspect, the computerized system may further include at least one camera arranged to capture at least one image of at least a portion of a user, or at least a portion of a physical object, or at least a portion of a user and at least a portion of a physical object. The at least one computer processor may determine information about the user's interaction with the physical object based at least in part on the at least one image.

[0009] In another aspect, at least one computer processor may instruct the AR system to provide feedback to the user.

[0010] In a variation of this aspect, at least one computer processor may instruct the AR system to provide the feedback as visual feedback to the user in the AR environment. In one example, at least one processor may instruct the AR system to provide the visual feedback to the user in the AR environment as a change in at least one property of at least one visual indicator displayed by the AR system. In another example, at least one processor may instruct the AR system to provide the visual feedback to the user in the AR environment as a display of at least one new visual indicator associated with a physical object in the AR environment.

[0011] In another variation of this aspect, at least one processor may instruct the AR system to provide feedback to the user as audio feedback, or electrical stimulation feedback, or both.

[0012] In a further variation of this aspect, the feedback provided to the user may include an indication of an amount of force applied by the user to the physical object. The amount of force may be determined based at least in part on the plurality of neuromuscular signals. In one example, the computerized system may further include at least one camera arranged to capture at least one image; the amount of force applied to the physical object may be determined based at least in part on the at least one image.

[0013] In one aspect, when at least one function of the physical object is used in the AR environment, at least one computer processor may instruct the AR system to provide feedback as a change to the at least one function.

[0014] In another aspect, at least one computer processor may instruct the AR system to provide feedback to a different user other than the user interacting with the physical object.

[0015] In variations of this aspect, feedback may be provided to different users in an AR environment that they experience differently.

[0016] In one aspect, the computerized system may further include a tactile circuit arranged to transmit a tactile signal to the user. The tactile circuit may be arranged on a wearable device worn by the user. The at least one computer processor may be programmed to instruct the AR system or a controller external to the AR system to provide feedback to the user as tactile feedback transmitted via the tactile circuit.

[0017] In variations of this aspect, the haptic circuit may include any one or any combination of a vibration actuator, a skin tap actuator, a low voltage electric shock stimulation circuit, and a force actuator.

[0018] In another variation of this aspect, the wearable device on which the haptic circuit is disposed may include a wearable patch.

[0019] In another variation of this aspect, the haptic circuit may be disposed on one or more wearable devices, with a plurality of neuromuscular sensors disposed on the wearable device.

[0020] In another aspect, the information about the user's interaction with the physical object may include information that the user has interacted with a particular physical object. The at least one computer processor may instruct the AR system to provide feedback to the user as a modification of at least one interaction property of the particular physical object and as an indication of the modification.

[0021] In variations of this aspect, the modification of at least one interactive property of the particular physical object may include enabling at least one virtual control associated with the particular physical object. In further variations, the at least one computer processor may be programmed to disable at least one virtual control associated with the particular physical object in response to input received from a user. In further variations, the at least one virtual control associated with the particular physical object may be disabled by the at least one computer processor based at least in part on a plurality of neuromuscular signals.

[0022] In other variations of this aspect, the particular physical object may include a writing instrument. The modification of at least one interactive attribute may include activating a set of enhanced features of the writing instrument. In one example, the set of enhanced features may include features that enable a user to interact with the writing instrument in one or more ways to change one or more writing characteristics of the writing instrument in the AR environment. In another example, at least one computer processor may be programmed to: determine, based at least in part on a plurality of neuromuscular signals, that the user is interacting with the writing instrument in one of one or more ways; and in response to determining that the user is interacting with the writing instrument in one of one or more ways, change the corresponding writing characteristics of the writing instrument. One or more writing characteristics may include any one or any combination of the following: writing color, line thickness, brush shape, drawing mode, and erasing mode.

[0023] In one aspect, the functionality of a physical object in an AR environment can be controlled by a set of virtual controls. The at least one processor can be programmed to instruct the AR system to perform any one or any combination of the following operations based at least in part on the plurality of neuromuscular signals: activate the set of virtual controls, deactivate the set of virtual controls, and modify the set of virtual controls.

[0024] In a variation of this aspect, the plurality of neuromuscular signals may include signals generated by a gesture performed by the user. The gesture may include any one or any combination of the following: a static gesture, a dynamic gesture, a hidden gesture, a muscle activation state, and a muscle under-activation state.

[0025] In another aspect, the AR environment may include a plurality of physical objects. Each of the plurality of physical objects may be associated with a set of control actions, and at least one computer processor may be programmed to: identify the physical object with which the user interacts in the AR environment, and activate a corresponding set of control actions associated with the identified physical object.

[0026] In a variation of this aspect, the feedback may include a visual display in the AR environment. The visual display may indicate a set of control actions that are activated.

[0027] In another variation of this aspect, the computerized system may further include at least one camera arranged to capture at least one image. The at least one computer processor may identify a physical object with which the user is interacting in the AR environment from a plurality of physical objects in the AR environment based at least in part on the at least one image captured by the at least one camera. The at least one computer processor may be programmed to identify that the user is interacting with the identified physical object based at least in part on the plurality of neuromuscular signals.

[0028] In another variation of this aspect, at least one computer processor may be programmed to determine information about the user's interaction with the physical object based at least in part on the activated set of control actions.

[0029] In another variation of this aspect, at least one computer processor may determine information about a user's interaction with a physical object based at least in part on an output of an inference model, wherein a plurality of neuromuscular signals or information derived from a plurality of neuromuscular signals, or a plurality of neuromuscular signals and information derived from a plurality of neuromuscular signals are provided as input to the inference model. Prior to rendering the output of the inference model, the at least one computer processor may provide the inference model with information about the identified physical object, or information associated with the set of control actions activated, or information about the identified physical object and information associated with the set of control actions activated.

[0030] In one aspect, the AR environment may include a plurality of physical objects. Each of the plurality of physical objects may be associated with a set of control actions. At least one computer processor may be programmed to instruct the AR system to: select a particular physical object from the plurality of physical objects for active control in the AR environment based at least in part on the plurality of neuromuscular signals; and activate a corresponding set of control actions associated with the particular physical object.

[0031] In a variation of this aspect, the plurality of neuromuscular signals may include signals generated by a gesture performed by the user. The gesture may include any one or any combination of the following: a static gesture, a dynamic gesture, a hidden gesture, a muscle activation state, and a muscle under-activation state.

[0032] In another aspect, at least one computer processor may be programmed to modify operation of the AR system based at least in part on information regarding user interactions with physical objects.

[0033] In variations of this aspect, at least one computer processor may modify operation of the AR system by instructing the AR system to enter a higher precision mode to detect finer-grained interactions of the user with the physical object. In one example, while in the higher precision mode, the AR system may use a greater weight for the plurality of neuromuscular signals than for auxiliary signals from at least one auxiliary sensor to determine information about the user's interaction with the physical object.

[0034] According to aspects of the technology described herein, a method is provided, wherein the method is performed by a computerized system to enable a user to interact with a physical object in an augmented reality (AR) environment generated by an AR system based at least in part on neuromuscular signals. The method may include: using a plurality of neuromuscular sensors arranged on one or more wearable devices worn by the user to sense the plurality of neuromuscular signals, to obtain a plurality of neuromuscular signals from the user; using at least one computer processor coupled to a memory storing code executed by at least one computer processor, determining information about the user's interaction with the physical object in the AR environment generated by the AR system based at least in part on the plurality of neuromuscular signals; and using at least one computer processor, instructing the AR system to provide feedback based at least in part on the information about the user's interaction with the physical object.

[0035] In one aspect, the method may further include instructing, using at least one computer processor, the AR system to display at least one visual indicator on a physical object in the AR environment.

[0036] In variations of this aspect, the determination of information regarding the user's interaction with the physical object may be based at least in part on the user's interaction with at least one visual indicator displayed on the physical object.

[0037] In another aspect, the method can further include capturing, using at least one camera, at least one image of at least a portion of the user, or at least a portion of the physical object, or at least a portion of the user and at least a portion of the physical object. The determination of information about the user's interaction with the physical object can be based at least in part on the at least one image.

[0038] In one aspect, the indication may instruct the AR system to provide feedback to the user.

[0039] In a variation of this aspect, the indication may instruct the AR system to provide the feedback as visual feedback to the user in the AR environment. In one example, the indication may instruct the AR system to provide the visual feedback to the user in the AR environment as a change in at least one property of at least one visual indicator displayed by the AR system. In another example, the visual feedback is provided to the user in the AR environment as a display of at least one new visual indicator associated with a physical object in the AR environment.

[0040] In another variation, the instruction may instruct the AR system to provide feedback to the user as audio feedback, or electrical stimulation feedback, or both.

[0041] In another variation, the feedback provided to the user may include an indication of an amount of force applied by the user to the physical object. The amount of force may be determined based at least in part on the plurality of neuromuscular signals. In one example, the method may further include capturing at least one image using at least one camera; the amount of force applied to the physical object may be determined based at least in part on the at least one image.

[0042] On the other hand, when at least one function of the physical object is used in the AR environment, the indication may instruct the AR system to provide the feedback as a change to the at least one function.

[0043] In one aspect, the indication may instruct the AR system to provide feedback to a different user than the user interacting with the physical object.

[0044] In variations of this aspect, feedback may be provided to different users in an AR environment that they experience differently.

[0045] In another aspect, the method may further include, using at least one computer processor, instructing the AR system or a controller external to the AR system to provide feedback to the user as tactile feedback delivered via a tactile circuit arranged to deliver tactile signals to the user. The tactile circuit may be arranged on a wearable device worn by the user.

[0046] In variations of this aspect, the haptic circuit may include any one or any combination of a vibration actuator, a skin tap actuator, a low voltage electric shock stimulation circuit, and a force actuator.

[0047] In another variation of this aspect, the wearable device on which the haptic circuit is disposed may include a wearable patch.

[0048] In another variation, the haptic circuit may be disposed on one or more wearable devices, with a plurality of neuromuscular sensors disposed on the wearable device.

[0049] In one aspect, the information about the user's interaction with the physical object may include information that the user has interacted with a particular physical object. The indication may instruct the AR system to provide feedback to the user as a modification of at least one interaction property of the particular physical object and as an indication of the modification.

[0050] In a variation of this aspect, the modification of at least one interactive property of the particular physical object may include enabling at least one virtual control associated with the particular physical object. In another variation, the method may also include disabling, by at least one computer processor, at least one virtual control associated with the particular physical object in response to input from a user. In one example, at least one virtual control associated with the particular physical object may be disabled by at least one computer processor based at least in part on the plurality of neuromuscular signals.

[0051] In another variation of this aspect, the particular physical object may include a writing instrument. The modification of at least one interactive attribute may include activating a set of enhanced features of the writing instrument. In one example, the set of enhanced features may include features that enable a user to interact with the writing instrument in one or more ways to change one or more writing characteristics of the writing instrument in the AR environment. In a further variation, the method may further include: determining by at least one computer processor that the user is interacting with the writing instrument in one of the one or more ways based at least in part on a plurality of neuromuscular signals; and in response to determining that the user is interacting with the writing instrument in one of the one or more ways, changing the corresponding writing characteristics of the writing instrument by at least one computer processor. One or more writing characteristics may include any one or any combination of the following: writing color, line thickness, brush shape, drawing mode, and erasing mode.

[0052] In another aspect, the functionality of a physical object in an AR environment can be controlled by a set of virtual controls. The method can further include using at least one computer processor, based at least in part on the plurality of neuromuscular signals, to instruct the AR system to perform any one or any combination of the following operations: activate the set of virtual controls, deactivate the set of virtual controls, and modify the set of virtual controls.

[0053] In a variation of this aspect, the plurality of neuromuscular signals may include signals generated by a gesture performed by the user. The gesture may include any one or any combination of the following: a static gesture, a dynamic gesture, a hidden gesture, a muscle activation state, and a muscle under-activation state.

[0054] In one aspect, the AR environment may include a plurality of physical objects. Each of the plurality of physical objects may be associated with a set of control actions. The method may also include: identifying, by at least one computer processor, a physical object with which a user interacts in the AR environment; and activating, by at least one computer processor, a corresponding set of control actions associated with the identified physical object.

[0055] In a variation of this aspect, the feedback may include a visual display in the AR environment. The visual display may indicate a set of control actions that are activated.

[0056] In another variation of this aspect, the method may further include capturing at least one image using at least one camera. The recognition may identify the physical object from a plurality of physical objects in the AR environment based at least in part on the at least one image captured by the at least one camera. The at least one computer processor may recognize that the user is interacting with the identified physical object based at least in part on the plurality of neuromuscular signals.

[0057] In another variation of this aspect, the determination of information regarding the user's interaction with the physical object may be based at least in part on an activated set of control actions.

[0058] In another variation of this aspect, the determination of information about the user's interaction with the physical object can be based at least in part on the output of the reasoning model, and the plurality of neuromuscular signals or information derived from the plurality of neuromuscular signals, or the plurality of neuromuscular signals and information derived from the plurality of neuromuscular signals are provided as input to the reasoning model. The method can also include, before rendering the output of the reasoning model, providing the reasoning model with information about the identified physical object, or information associated with the activated set of control actions, or information about the identified physical object and information associated with the activated set of control actions.

[0059] In another aspect, the AR environment may include a plurality of physical objects. Each of the plurality of physical objects may be associated with a set of control actions. The method may also include instructing the AR system, using at least one computer processor, to: select a particular physical object from the plurality of physical objects for active control in the AR environment based at least in part on the plurality of neuromuscular signals; and activate a corresponding set of control actions associated with the particular physical object.

[0060] In a variation of this aspect, the plurality of neuromuscular signals may include signals generated by a gesture performed by the user. The gesture may include any one or any combination of the following: a static gesture, a dynamic gesture, a hidden gesture, a muscle activation state, and a muscle under-activation state.

[0061] In one aspect, the method may further include modifying, by the at least one computer processor, operation of the AR system based at least in part on the information regarding the user's interaction with the physical object.

[0062] In a variation of this aspect, at least one computer processor may modify the operation of the AR system by instructing the AR system to enter a higher precision mode to detect finer-grained interactions of the user with the physical object. In another variation, when in the higher precision mode, the AR system may use a greater weight for the plurality of neuromuscular signals than for the auxiliary signals from at least one auxiliary sensor to determine information about the user's interaction with the physical object.

[0063] According to various aspects of the technology described herein, a non-transitory computer-readable medium is provided, wherein a plurality of instructions are encoded, and when executed by at least one computer processor, the instructions cause the at least one computer processor to perform a method for enabling a user to interact with a physical object in an augmented reality (AR) environment generated by an AR system at least partially based on neuromuscular signals. The method may include: receiving a plurality of neuromuscular signals obtained from a user via a plurality of neuromuscular sensors as input, the plurality of neuromuscular sensors being arranged on one or more wearable devices worn by the user to sense the plurality of neuromuscular signals; determining information about the user's interaction with a physical object in an AR environment generated by the AR system based at least in part on the plurality of neuromuscular signals; and instructing the AR system to provide feedback based at least in part on the information about the user's interaction with the physical object.

[0064] In one aspect, the method may further include instructing the AR system to display at least one visual indicator on a physical object in the AR environment.

[0065] In variations of this aspect, the determination of information regarding the user's interaction with the physical object may be based at least in part on the user's interaction with at least one visual indicator displayed on the physical object.

[0066] In another aspect, the method may further include receiving as input from at least one camera at least one image of at least a portion of a user or at least a portion of a physical object or at least a portion of a user and at least a portion of a physical object. The determination of information about the user's interaction with the physical object may be based at least in part on the at least one image.

[0067] In one aspect, the indication may instruct the AR system to provide feedback to the user.

[0068] In a variation of this aspect, the indication may instruct the AR system to provide the feedback as visual feedback to the user in the AR environment. In one example, the indication may instruct the AR system to provide the feedback to the user in the AR environment as a change in at least one property of at least one visual indicator displayed by the AR system. In another example, the visual feedback may be provided to the user in the AR environment as a display of at least one new visual indicator associated with a physical object in the AR environment.

[0069] In another variation, the instruction may instruct the AR system to provide feedback to the user as audio feedback, or electrical stimulation feedback, or both.

[0070] In another variation, the feedback provided to the user may include an indication of an amount of force applied by the user to the physical object. The amount of force may be determined based at least in part on the plurality of neuromuscular signals. In one example, the method may further include receiving at least one image as input from at least one camera; the amount of force applied to the physical object may be determined based at least in part on the at least one image.

[0071] On the other hand, when at least one function of the physical object is used in the AR environment, the indication may instruct the AR system to provide the feedback as a change to the at least one function.

[0072] In one aspect, the indication may instruct the AR system to provide feedback to a different user other than the user interacting with the physical object.

[0073] In variations of this aspect, feedback may be provided to different users in an AR environment that they experience differently.

[0074] In another aspect, the method may further include instructing the AR system or a controller external to the AR system to provide feedback to the user as tactile feedback delivered via a tactile circuit arranged to deliver a tactile signal to the user. The tactile circuit may be arranged on a wearable device worn by the user.

[0075] In variations of this aspect, the haptic circuit may include any one or any combination of a vibration actuator, a skin tap actuator, a low voltage electric shock stimulation circuit, and a force actuator.

[0076] In another variation of this aspect, the wearable device on which the haptic circuit is disposed may include a wearable patch.

[0077] In another variation of this aspect, the haptic circuit may be disposed on one or more wearable devices, with a plurality of neuromuscular sensors disposed on the wearable device.

[0078] In one aspect, the information about the user's interaction with the physical object may include information that the user has interacted with a particular physical object. The indication may instruct the AR system to provide feedback to the user as a modification of at least one interaction property of the particular physical object and as an indication of the modification.

[0079] In variations of this aspect, the modification of at least one interactive property of the particular physical object may include enabling at least one virtual control associated with the particular physical object. In further variations, the method may further include disabling at least one virtual control associated with the particular physical object in response to input from the user. In one example, based at least in part on the plurality of neuromuscular signals, the at least one virtual control associated with the particular physical object may be disabled by at least one computer processor.

[0080] In another variation of this aspect, the particular physical object may include a writing instrument. The modification of at least one interactive attribute may include activating a set of enhanced features of the writing instrument. In a further variation, the set of enhanced features may include features that enable a user to interact with the writing instrument in one or more ways to change one or more writing characteristics of the writing instrument in the AR environment. In a further variation, the method may further include: determining that the user is interacting with the writing instrument in one of the one or more ways based at least in part on a plurality of neuromuscular signals; and in response to determining that the user is interacting with the writing instrument in one of the one or more ways, changing the corresponding writing characteristics of the writing instrument. The one or more writing characteristics may include any one or any combination of the following: writing color, line thickness, brush shape, drawing mode, and erasing mode.

[0081] In another aspect, the functionality of a physical object in an AR environment can be controlled by a set of virtual controls. The method can further include using at least one, based at least in part on the plurality of neuromuscular signals, to instruct the AR system to perform any one or any combination of the following: activate the set of virtual controls, deactivate the set of virtual controls, and modify the set of virtual controls.

[0082] In a variation of this aspect, the plurality of neuromuscular signals may include signals generated by a gesture performed by the user. The gesture may include any one or any combination of the following: a static gesture, a dynamic gesture, a hidden gesture, a muscle activation state, and a muscle under-activation state.

[0083] In one aspect, the AR environment may include a plurality of physical objects. Each of the plurality of physical objects may be associated with a set of control actions. The method may also include: identifying a physical object with which the user interacts in the AR environment; and activating a corresponding set of control actions associated with the identified physical object.

[0084] In a variation of this aspect, the feedback may include a visual display in the AR environment. The visual display may indicate a set of control actions that are activated.

[0085] In another variation of this aspect, the method may further include capturing at least one image using at least one camera. The recognition may identify the physical object from a plurality of physical objects in the AR environment based at least in part on the at least one image captured by the at least one camera. Based at least in part on the plurality of neuromuscular signals, it may be determined that the user is interacting with the identified physical object.

[0086] In another variation of this aspect, the determination of information regarding the user's interaction with the physical object may be based at least in part on an activated set of control actions.

[0087] In another variation of this aspect, the determination of information about the user's interaction with the physical object can be based at least in part on the output of the reasoning model, and the plurality of neuromuscular signals or information derived from the plurality of neuromuscular signals, or the plurality of neuromuscular signals and information derived from the plurality of neuromuscular signals are provided as input to the reasoning model. The method can also include, before rendering the output of the reasoning model, providing the reasoning model with information about the identified physical object, or information associated with the activated set of control actions, or information about the identified physical object and information associated with the activated set of control actions.

[0088] In another aspect, the AR environment may include a plurality of physical objects. Each of the plurality of physical objects may be associated with a set of control actions. The method may further include instructing the AR system to: select a particular physical object from the plurality of physical objects for active control in the AR environment based at least in part on the plurality of neuromuscular signals; and activate a corresponding set of control actions associated with the particular physical object.

[0089] In a variation of this aspect, the plurality of neuromuscular signals may include signals generated by a gesture performed by the user. The gesture may include any one or any combination of the following: a static gesture, a dynamic gesture, a hidden gesture, a muscle activation state, and a muscle under-activation state.

[0090] In one aspect, the method may further include modifying operation of the AR system based at least in part on the information regarding the user's interaction with the physical object.

[0091] In a variation of this aspect, the modification to the operation of the AR system may include instructing the AR system to enter a higher precision mode to detect finer-grained interactions of the user with the physical object. In another variation, when in the higher precision mode, the AR system may use a greater weight for the plurality of neuromuscular signals than for the auxiliary signals from at least one auxiliary sensor to determine information about the user's interaction with the physical object.

[0092] According to aspects of the technology described herein, a kit for controlling an augmented reality (AR) system is provided. The kit may include: a wearable device including a plurality of neuromuscular sensors configured to sense a plurality of neuromuscular signals of a user; and a non-transitory computer-readable medium encoded with a plurality of instructions, which, when executed by at least one computer processor, causes the at least one computer processor to perform a method for enabling a user to interact with a physical object in an AR environment generated by the AR system. The method may include: receiving a plurality of neuromuscular signals sensed from a user by a plurality of neuromuscular sensors as input; determining information about an interaction of a user with a physical object in an AR environment generated by the AR system based at least in part on the plurality of neuromuscular signals; and instructing the AR system to provide feedback based at least in part on the information about the interaction of the user with the physical object.

[0093] In one aspect, a wearable device may include a wearable band configured to be worn about a part of a user.

[0094] In another aspect, a wearable device may include a wearable patch configured to be worn on a part of a user.

[0095] It should be appreciated that all combinations of the foregoing concepts and additional concepts discussed in more detail below (provided these concepts are not mutually inconsistent) are considered to be part of the inventive subject matter disclosed herein. In particular, all combinations of the claimed subject matter appearing in this disclosure are considered to be part of the inventive subject matter disclosed herein. BRIEF DESCRIPTION OF THE DRAWINGS

[0096] Various non-limiting embodiments of the present technology will be described with reference to the following drawings. It should be understood that the drawings are not necessarily drawn to scale.

[0097] Figure 1 is a schematic diagram of a computer-based system for processing neuromuscular sensor data, such as signals obtained from a neuromuscular sensor, according to some embodiments of the technology described herein; Figure 2 is a schematic diagram of a distributed computer-based system integrating an AR system with a neuromuscular activity system according to some embodiments of the technology described herein; Figure 3 is a flow chart of a process for providing an enhanced AR experience using neuromuscular signals according to some embodiments of the technology described herein; Figure 4 is a flowchart of a process for providing virtual controls for physical objects in an AR environment according to some embodiments of the technology described herein; Figure 5is a flow chart of a process for activating a set of control actions for physical objects in an AR environment according to some embodiments of the technology described herein; Fig. 6A , Figure 6B , Figure 6C and Fig.6D schematically illustrates a patch-type wearable system with sensor electronics incorporated thereon according to some embodiments of the technology described herein; Fig. 7A A wristband having EMG sensors circumferentially arranged thereon is shown according to some embodiments of the technology described herein; Figure 7B The user is shown wearing the Fig. 7A wristband; Fig. 8A shows a wearable system having 16 EMG sensors arranged circumferentially around a band configured to be worn around a user's lower arm or wrist according to some embodiments of the technology described herein; Figure 8B yes Fig. 8A A cross-sectional view of one of the 16 EMG sensors shown in FIG. Fig. 9A and Fig. 9B Components of a computer-based system are schematically shown in which some embodiments of the techniques described herein are implemented. Fig. 9A The wearable portion of the computer-based system is shown, and Fig. 9B A dongle portion connected to a computer is shown, wherein the dongle portion is configured to communicate with the wearable portion.

[0098] Fig.10 is a diagram schematically illustrating an example of an implementation using an EMG sensor and a camera according to some embodiments of the technology described herein. Detailed Description

[0099] The inventors have developed new techniques for controlling AR systems, as well as other types of XR systems such as VR systems and MR systems. Various embodiments of the techniques presented herein provide certain advantages, including avoiding the use of undesirable or cumbersome physical keyboards or microphones; overcoming problems associated with time-consuming and / or high-latency processing of low-quality images of a user captured by a camera; allowing capture and detection of subtle, minute, or rapid movements and / or pressure changes on an object (e.g., varying amounts of force applied by a stylus, writing implement, or finger pressing against a surface), which may be important for solving, for example, text input; collecting and analyzing various sensory information that enhances the control recognition process, which may not be easily accomplished when using traditional input devices; and enabling hand-based control in situations where a user's hands are occluded or outside the camera's field of view (e.g., in a user's pocket, or when the user is wearing gloves).

[0100] Some embodiments of the technology described herein relate to coupling a system for sensing neuromuscular signals through neuromuscular sensors worn by a user with a system for performing AR functions. In particular, a neuromuscular system that senses neuromuscular signals in order to determine the positioning of a body part (e.g., a hand, an arm, etc.) can be used in conjunction with an AR system to provide an improved AR experience for the user. For example, information obtained in both systems can be used to improve the overall AR experience. The AR system can include a camera to capture image information about one or more body parts of the user, and the image information can be used to improve the user's interaction with the AR environment generated by the AR system. For example, a musculoskeletal representation associated with one or more body parts of the user can be generated based on sensor data from a neuromuscular sensor, and the sensor data can be supplemented with image data of the user captured by a camera in the AR system, for example, to achieve a more realistic visualization of the user relative to one or more objects in the AR environment. In one embodiment of this example, the image data of the user can be used to determine an object of interest to the user, and the sensor data can provide muscle activation information, which is used to determine the type of action to be performed relative to the object and / or the amount of force to be used for the action (e.g., gently push the object, push the object hard, tap the object, etc.). In another embodiment, the displayed information in the AR environment can be used as feedback to the user to allow the user to more precisely control his / her musculoskeletal input (eg, motor input) to the neuromuscular system.

[0101] The inventors recognize that neither cameras nor neuromuscular sensors are ideal input systems by themselves. Cameras such as those that may be provided in AR systems can provide good positioning information (relative to other skeletal parts and external objects) when, for example, a user's joint parts are clearly within the field of view, but may be limited by field of view limitations and occlusions and may not be suitable for measuring forces. At the same time, the signals measured or detected by the neuromuscular sensors (e.g., electromyography (EMG) signals or another modality of neuromuscular signals as described herein) may not themselves be sufficient to distinguish between the forces applied by the user to himself / herself and the forces applied by him / her to external objects, and such signals may not provide sufficiently accurate information about skeletal geometry, such as finger length. According to some embodiments, it should be understood that it would be beneficial to increase the accuracy of AR systems and neuromuscular sensor-based systems to provide a more accurate and realistic user experience.

[0102] Some conventional AR systems include camera-based technologies for identifying and mapping physical objects in the user's real-world environment. In an AR environment generated by an AR system, such camera-based technologies are generally insufficient to measure and implement a full range of possible physical and virtual interactions with physical objects. To this end, some embodiments of the technology described herein are directed to an AR-based system that includes an improved AR system that provides a rich AR user experience by interpreting neuromuscular signals obtained via a wearable neuromuscular sensor device worn by a user of an AR-based system. In some embodiments, the motion activity state determined from the neuromuscular signal can be used to determine whether and how the user interacts with the physical object in the AR environment. In other embodiments, the motion activity state determined from the neuromuscular signal can be used to change the mode of the AR system, for example, by activating a set of control actions of the physical object in response to the determined motion activity state, turning the physical object into one or more "enhanced" objects. In various embodiments, when a user interacts with a physical object in an AR environment, a visual indicator based on the user's neuromuscular signal can be used to improve the user experience. Further examples of using neuromuscular signals to enhance interactions with physical objects in an AR environment are described in more detail below.

[0103] It should be understood that although various embodiments may be described herein with reference to AR-based systems, the scope of the technology disclosed herein enables these embodiments to be implemented using other types of XR-based systems.

[0104] According to some embodiments of the technology disclosed herein, neuromuscular signals sensed and recorded by one or more wearable sensors can be used to determine information about a user's interaction or expected interaction with a physical object in an AR environment generated by an AR-based system. Such signals may also be referred to herein as "sensing signals". The sensed signals can be used directly as inputs to the AR system (e.g., by using motor unit action potentials as input signals), and / or the sensed signals can be processed (including by using the inference model described herein) to determine the movement, force, and / or positioning of a part of the user's body (e.g., a finger, hand, wrist, etc.). For example, a neuromuscular signal obtained by a neuromuscular sensor arranged on a wearable device can be used to determine the force (e.g., grip force) applied to a physical object. The inventors have recognized that multiple muscle activation states of a user can be identified from the sensing signals and / or information based on the sensing signals to provide an improved AR experience. Muscle activation states may include, but are not limited to, static gestures or postures performed by a user, dynamic gestures or actions performed by a user, a muscle activation state of a user, muscle tension or relaxation performed by a user, or any combination of the foregoing. As described herein, the user's interaction with one or more physical objects in the AR environment can take a variety of forms, including but not limited to: selecting one or more objects, controlling one or more objects, activating or deactivating one or more objects, adjusting settings or features related to one or more objects, etc. It should be understood that the user's interaction can take other forms implemented by the AR system for the environment, and need not be the interactions specifically listed herein. For example, the control performed in the AR environment can include control based on the activation of one or more individual motor units, such as control based on the detected activation state of the user's muscles (such as sensed muscle tension). It should be understood that when used in conjunction with sensor signals from neuromuscular sensors, the phrases "sensing", "obtaining", "collecting", "sensing and recording", "measuring", "recording", etc. include signals detected by the sensor. It should be understood that the signal can be recorded, or sensed and recorded without being stored in non-volatile memory, or the signal can be recorded, or sensed and recorded, while being stored in local non-volatile memory or external non-volatile memory. For example, after detection, the signal may be stored at the sensor "as detected" (i.e., raw), or the signal may be processed at the sensor before being stored at the sensor, or the signal may be transmitted (e.g., via Bluetooth technology, etc.) to an external device for processing and / or storage, or any combination of the foregoing.

[0105] Identification of one or more muscle activation states may allow for a hierarchical or multi-level approach to interacting with physical objects in an AR environment. For example, at a first layer / level, one muscle activation state may indicate that a user is interacting with a physical object; at a second layer / level, another muscle activation state may indicate that a user wants to activate a set of virtual controls and / or features for a physical object in an AR environment that is interacting with them; at a third layer / level, yet another muscle activation state may indicate which activated virtual controls and / or features the user wants to use when interacting with the object. It should be understood that any number of muscle activation states and layers may be used without departing from the scope of the present disclosure. For example, in some embodiments, one or more muscle activation states may correspond to concurrent gestures based on the activation of one or more motor units, such as a user's hand bending at the wrist while an index finger points at an object. In some embodiments, one or more muscle activation states may correspond to a sequence of gestures based on the activation of one or more motor units, such as a user's hand grabbing an object and lifting the object. In some embodiments, a single muscle activation state may indicate both a user's desire to interact with a physical object and the activation of a set of virtual controls and / or features for interacting with an object.

[0106] As an example, sensor signals may be sensed and recorded for a first activity of a user (e.g., a first gesture performed by the user), and a first muscle activation state of the user may be identified from these sensed signals using, for example, a trained inference model, as described below. The first muscle activation state may indicate that the user is interacting with a particular physical object (e.g., a writing instrument) in the user's environment. In response to the system detecting the first activity, feedback may be provided to identify the interaction with the physical object indicated by the first muscle activation state. Examples of the types of feedback that may be provided according to some embodiments of the present technology are discussed in more detail below. The sensor signals may continue to be sensed and recorded, and a second muscle activation state may be determined. In response to identifying the second muscle activation state (e.g., corresponding to a second gesture that may be the same or different from the first gesture), the AR system may activate a set of virtual controls for the object (e.g., controls for selecting writing characteristics of a writing instrument). The sensor signals may continue to be sensed and recorded, and a third muscle activation state may be determined. The third muscle activation state may indicate a selection from the virtual controls. For example, the third muscle activation state may indicate a selection of a particular line thickness of a writing instrument.

[0107] According to some embodiments, a muscle activation state may be identified at least in part from raw (e.g., unprocessed) sensor signals collected by one or more wearable sensors. In some embodiments, a muscle activation state may be identified at least in part from information based on raw sensor signals (e.g., processed sensor signals), where the raw sensor signals collected by one or more wearable sensors are processed to perform, for example, amplification, filtering, rectification, and / or other forms of signal processing, examples of which are described in more detail below. In some embodiments, a muscle activation state may be identified at least in part from an output of a trained inference model that receives as input a sensor signal (either a raw or processed version of the sensor signal).

[0108] As disclosed herein, muscle activation states determined based on sensor signals according to one or more of the techniques described herein can be used to interact with one or more physical objects in an AR environment without relying on cumbersome and inefficient input devices, as described above. For example, sensor data (e.g., signals obtained from neuromuscular sensors or data derived from these signals) can be sensed and recorded, and muscle activation states can be identified from the sensor data without requiring the user to carry a controller and / or other input device, and without requiring the user to remember complex button or key operation sequences. In addition, the identification of muscle activation states (e.g., gestures, postures, etc.) based on sensor data can be performed relatively quickly, thereby reducing response time and latency associated with issuing control signals to the AR system. In addition, some embodiments of the techniques described herein allow for user customization of an AR-based system, such that each user can define a control scheme for interacting with physical objects in an AR environment of an AR-based system, which is generally not possible with conventional AR systems.

[0109] Signals sensed by wearable sensors placed at locations on the user's body can be provided as input to an inference model that is trained to generate spatial information of rigid segments of a multi-segment articulated rigid body model of a human body. The spatial information may include, for example, positioning information of one or more segments, orientation information of one or more segments, joint angles between segments, and the like. Based on the input, and as a result of the training, the inference model can implicitly represent the inference motion of the articulated rigid body under defined motion constraints. The trained inference model can output data that can be used for an application, such as an application for rendering a representation of a user's body in an XR environment (e.g., the AR environment described above), where the user can interact with one or more physical and / or one or more virtual objects, and / or an application for monitoring user motion when the user performs physical activities to assess, for example, whether the user is performing physical activities in a desired manner. It should be understood that the output data from the trained inference model can be used for applications other than those specifically identified herein.

[0110] For example, motion data obtained by a single motion sensor positioned on a user (e.g., on the user's wrist or arm) can be provided as input data to a trained inference model. The corresponding output data generated by the trained inference model can be used to determine spatial information of one or more segments of a multi-segment articulated rigid body model for the user. For example, the output data can be used to determine the location and / or orientation of one or more segments in the multi-segment articulated rigid body model. In another example, the output data can be used to determine the angles between connected segments in the multi-segment articulated rigid body model.

[0111] It should be understood that the inference model used in conjunction with neuromuscular signals may relate to generalized skeletal geometry for a class of users (e.g., a typical adult male, a typical child, a typical adult female), or may relate to user-specific skeletal geometry for a particular user.

[0112] Different types of sensors can be used to provide input data to the trained inference model, as described below.

[0113] As briefly described herein, in some embodiments of the present technology, various muscle activation states can be identified directly from sensor data. In other embodiments, hand states, gestures, postures, etc. (which may be referred to herein as muscle activation states, individually or collectively) can be identified based at least in part on the output of a trained inference model. In some embodiments, the trained inference model can output motor units or muscle activations and / or positioning, orientation, and / or force estimates for segments of a computer-generated musculoskeletal model. In one embodiment, all or part of the human musculoskeletal system can be modeled as a multi-segment articulated rigid body system in which joints form interfaces between different segments and the joint angles define the spatial relationships between connected segments in the model.

[0114] As used herein, the term "gesture" may refer to a static or dynamic configuration of one or more body parts, including the positioning of one or more body parts and the forces associated with the configuration. For example, a gesture may include a discrete gesture (such as placing or pressing the palm down on a solid surface or grabbing a ball), a continuous gesture (such as waving fingers back and forth, grabbing and throwing a ball), or a combination of discrete and continuous gestures. Gestures may include covert gestures that may not be perceived by others, such as slightly tightening a joint by co-contracting opposing muscles or using submuscular activation. When training an inference model, gestures may be defined using an application that is configured to prompt the user to perform gestures, or gestures may be arbitrarily defined by the user. Gestures performed by the user may include symbolic gestures (e.g., gestures mapped to other gestures, interactions, or commands based on a gesture vocabulary that specifies mappings). In some cases, hand gestures and arm gestures may be symbolic and used for communication according to cultural standards.

[0115] In some embodiments of the technology described herein, sensor signals may be used to predict information about the positioning and / or motion of a portion of a user's arm and / or a user's hand (which may be represented as a multi-segment articulated rigid body system having joints connecting multiple segments of the rigid body system). For example, in the case of hand motion, signals sensed and recorded by wearable neuromuscular sensors placed at locations on the user's body (e.g., the user's arm and / or wrist) may be provided as input to an inference model that is trained to predict the positioning (e.g., absolute positioning, relative positioning, orientation) and force estimates associated with multiple rigid segments in a computer-based musculoskeletal representation associated with the hand as the user performs one or more hand motions. The combination of positioning information and force information associated with the segments of the musculoskeletal representation associated with the hand may be referred to herein as the "hand state" of the musculoskeletal representation. As the user performs different motions, the trained inference model may interpret the neuromuscular signals sensed and recorded by the wearable neuromuscular sensors as positioning and force estimates (hand state information) for updating the musculoskeletal representation. Because neuromuscular signals can be continuously sensed and recorded, the musculoskeletal representation can be updated in real time, and a visual representation of the hand can be rendered (e.g., in an AR environment) based on the current estimate of the hand state. It should be understood that the estimate of the user's hand state can be used to determine the gesture the user is performing and / or predict the gesture the user will perform.

[0116] The limits of motion at the joints are determined by the type of joints connecting the segments and the biological structures (e.g., muscles, tendons, ligaments) that may limit the range of motion at the joints. For example, the shoulder joint connecting the upper arm and torso of a human subject and the hip joint connecting the thigh and torso are ball-and-socket joints that allow extension and flexion motions as well as rotational motions. In contrast, the elbow joint connecting the upper arm and lower arm (or forearm) of a human subject and the knee joint connecting the thigh and lower leg of a human subject allow a more limited range of motion. In this example, a multi-segment articulated rigid body system can be used to model portions of the human musculoskeletal system. However, it should be understood that although some segments of the human musculoskeletal system (e.g., forearms) can be approximated as rigid bodies in an articulated rigid body system, these segments can each include multiple rigid structures (e.g., a forearm can include an ulna and a radius), which can enable more complex motions within the segments that are not explicitly considered by the rigid body model. Therefore, a model of an articulated rigid body system used with some embodiments of the technology described herein can include segments that represent combinations of body parts that are not strictly rigid bodies. It should be understood that physical models other than multi-segment articulated rigid body systems may be used to model portions of the human musculoskeletal system without departing from the scope of the present disclosure.

[0117] Continuing with the example above, in kinematics, a rigid body is an object that exhibits various motion properties (e.g., position, orientation, angular velocity, acceleration). Knowing the motion properties of one segment of a rigid body enables the motion properties of other segments of the rigid body to be determined based on constraints on how the segments are connected. For example, a hand can be modeled as a multi-segment articulated body, with the joints in the wrist and each finger forming interfaces between multiple segments in the model. In some embodiments, the motion of segments in a rigid body model can be simulated as an articulated rigid body system, where the position (e.g., actual position, relative position, or orientation) information of the segments relative to other segments in the model is predicted using a trained inference model.

[0118] For some embodiments of the technology described herein, the human body part approximated by the musculoskeletal representation is a hand or a combination of a hand with one or more arm segments. In the musculoskeletal representation, information used to describe the current state of the positional relationships between the segments, the force relationships of the segments or combinations of segments, and the muscle and motor unit activation relationships between the segments is referred to herein as the hand state of the musculoskeletal representation (see discussion above). However, it should be understood that the technology described herein is also applicable to musculoskeletal representations of body parts other than hands, including but not limited to arms, legs, feet, torso, neck, or any combination of the foregoing.

[0119] In addition to spatial (e.g., position and / or orientation) information, some embodiments are able to predict force information associated with one or more segments of the musculoskeletal representation. For example, linear forces or rotational (torque) forces applied by one or more segments can be estimated. Examples of linear forces include, but are not limited to, the force of a finger or hand pressing on a solid object such as a table, and the force applied when two segments (e.g., two fingers) are squeezed together. Examples of rotational forces include, but are not limited to, rotational forces generated when a segment such as a wrist or finger is twisted or bent relative to another segment. In some embodiments, the force information determined as part of the current hand state estimate includes one or more of squeezing force information, grip force information, or information about co-contraction forces between muscles represented by the musculoskeletal representation.

[0120] Turning now to the accompanying drawings, Figure 1 A system 100, such as a neuromuscular activity system, according to some embodiments of the technology described herein is schematically shown. The system 100 may include one or more sensors 110 configured to sense and record signals generated by activation of motor units within one or more parts of a human body. The sensor 110 may include one or more neuromuscular sensors configured to sense and record signals generated by neuromuscular activity in human skeletal muscles. The term "neuromuscular activity" as used herein refers to neural activation, muscle activation, muscle contraction, or any combination of neural activation, muscle activation, and muscle contraction of spinal motor neurons or units that innervate muscles. One or more neuromuscular sensors may include one or more electromyography (EMG) sensors, one or more myograph (MMG) sensors, one or more sonomyography (SMG) sensors, two or more types of EMG sensors, a combination of MMG sensors and SMG sensors, and / or one or more sensors of any suitable type capable of detecting neuromuscular signals. In some embodiments, information about the user's interaction with a physical object in an AR environment may be determined based on neuromuscular signals sensed by one or more neuromuscular sensors. As the user moves over time, spatial information (e.g., position and / or orientation information) and force information associated with the motion can be predicted based on the sensed neuromuscular signals. In some embodiments, one or more neuromuscular sensors can sense muscle activity associated with motion caused by an external object, such as motion of a hand being pushed by an external object.

[0121] The one or more sensors 110 may include one or more auxiliary sensors, such as one or more inertial measurement units or IMUs, which use, for example, accelerometers, gyroscopes, magnetometers, or any combination of one or more accelerometers, gyroscopes, and magnetometers to measure a combination of physical aspects of movement. In some embodiments, one or more IMUs may be used to sense information about the movement of a body part to which the IMU is attached, and information derived from the sensed IMU data (e.g., positioning and / or orientation information) may be tracked as the user moves over time. For example, as the user moves over time, one or more IMUs may be used to track the movement of a portion of the user's body near the user's torso (e.g., arms, legs) relative to the IMU.

[0122] In an embodiment including at least one IMU and one or more neuromuscular sensors, the IMU and neuromuscular sensors may be arranged to detect movement of different parts of the human body. For example, the IMU may be arranged to detect movement of one or more body segments close to the torso (e.g., movement of the upper arm), while the neuromuscular sensor may be arranged to detect motor unit activity within one or more body segments away from the torso (e.g., movement of the lower arm (forearm) or wrist). However, it should be understood that the sensors (i.e., IMU and neuromuscular sensors) may be arranged in any suitable manner, and the embodiments of the technology described herein are not limited to being based on a specific sensor arrangement. For example, in some embodiments, at least one IMU and multiple neuromuscular sensors may be co-located on a body segment to track motor unit activity and / or movement of the body segment using different types of measurements. In one embodiment, the IMU and multiple EMG sensors may be arranged on a wearable device configured to be worn around the user's lower arm or wrist. In such an arrangement, the IMU can be configured to track motion information (e.g., position and / or orientation) associated with one or more arm segments over time to determine, for example, whether a user has raised or lowered his / her arm, while the EMG sensor can be configured to determine finer or more subtle motion information and / or submuscular information associated with activation of muscles or submuscular structures in the muscles of the wrist and / or hand.

[0123] During the performance of a motor task, as muscle tension increases, the firing rate of active neurons increases, and additional neurons may become active, a process that can be referred to as motor unit recruitment. The pattern in which neurons become active and increase their firing rate is stereotyped, so the expected motor unit recruitment pattern can define the activity manifold associated with standard or normal movement. In some embodiments, the sensor signal can identify the activation of a single motor unit or a group of motor units that are "off-manifold" because the pattern of motor unit activation is different from the expected or typical motor unit recruitment pattern. This activation that is off the manifold may be referred to as "sub-muscular activation" or "activation of sub-muscular structures" in this article, where the sub-muscular structure refers to a single motor unit or a group of motor units associated with the activation of the off-manifold. Examples of recruitment patterns of motor units that are off the manifold include, but are not limited to, selectively activating high-threshold motor units without activating lower-threshold motor units that would normally be activated earlier in the recruitment order, and regulating the firing rate of motor units over a considerable range without regulating the activity of other neurons that would normally be co-regulated in a typical motor unit recruitment pattern. In some embodiments, one or more neuromuscular sensors may be arranged relative to a human body and used to sense submuscular activation in the absence of observable movement (i.e., in the absence of readily observable corresponding movement of the body). In accordance with some embodiments of the technology described herein, submuscular activation may be used, at least in part, to interact with physical objects in an AR environment.

[0124] Some or all of the sensors 110 may each include one or more sensing components configured to sense information about the user. In the case of an IMU, the sensing components of the IMU may include one or more: an accelerometer, a gyroscope, a magnetometer, or any combination thereof to measure or sense characteristics of body motion, examples of which include, but are not limited to, acceleration, angular velocity, and magnetic field around the body during body motion. In the case of a neuromuscular sensor, the sensing component may include, but is not limited to, an electrode that detects electrical potentials on the surface of the body (e.g., for an EMG sensor), a vibration sensor that measures vibrations on the surface of the skin (e.g., for an MMG sensor), an acoustic sensing component that measures ultrasonic signals generated by muscle activity (e.g., for an SMG sensor), or any combination thereof. Optionally, the sensor 110 may include any one or any combination of the following: a thermal sensor (e.g., a thermistor) that measures the user's skin temperature; a heart sensor that measures the user's pulse, heart rate, a humidity sensor that measures the user's sweating status, and the like. Exemplary sensors that may be used as part of one or more sensors 110 according to some embodiments of the technology disclosed herein are described in greater detail in U.S. Patent No. 10,409,371, entitled “METHODS AND APPARATUS FOR INFERRING USERINTENT BASED ON NEUROMUSCULAR SIGNALS,” which is incorporated herein by reference.

[0125] In some embodiments, the one or more sensors 110 may include a plurality of sensors 110, and at least some of the plurality of sensors 110 may be arranged as part of a wearable device configured to be worn on or around a portion of a user's body. For example, in one non-limiting example, the IMU and a plurality of neuromuscular sensors may be arranged circumferentially on an adjustable and / or elastic band, such as a wristband or armband configured to be worn around a user's wrist or arm, as described in more detail below. In some embodiments, a plurality of wearable devices (each including one or more IMUs and / or one or more neuromuscular sensors) may be used to determine information about a user's interaction with a physical object based on activation from underlying muscle structures and / or based on motion involving multiple parts of the body. Alternatively, at least some of the sensors 110 may be arranged on a wearable patch configured to be attached to a portion of a user's body. Figure 6A-6D Various types of wearable patches are shown. Fig. 6AA wearable patch 62 is shown where circuitry for electronic sensors may be printed on a flexible substrate configured to be attached to an arm, such as near a vein to sense blood flow in a user. The wearable patch 62 may be an RFID type patch that can wirelessly transmit sensed information when interrogated by an external device. Figure 6B A wearable patch 64 is shown in which the electronic sensors may be incorporated on a substrate configured to be worn on the forehead of a user, for example, to measure moisture from perspiration. The wearable patch 64 may include circuitry for wireless communication, or may include a connector configured to be connected to a cable (e.g., a cable attached to a helmet, a head-mounted display, or another external device). The wearable patch 64 may be configured to adhere to the forehead of the user, or to be held against the forehead of the user by, for example, a headband, a skullcap, or the like. Figure 6C A wearable patch 66 is shown in which circuitry for an electronic sensor may be printed on a substrate configured to be adhered to a user's neck, for example, near the user's carotid artery to sense blood flow to the user's brain. The wearable patch 66 may be an RFID type patch or may include a connector configured to connect to an external electronic device. Fig.6D A wearable patch 68 is shown in which the electronic sensor may be incorporated into a substrate configured to be worn near a user's heart, such as to measure the user's heart rate or to measure blood flow to / from the user's heart. It should be understood that wireless communication is not limited to RFID technology and other communication technologies may be used. Likewise, it should be understood that the sensor 110 may be incorporated into other types of wearable patches that may be configured differently from the user's heart. Figure 6A-6D Those shown in .

[0126] In one embodiment, the sensor 110 may include sixteen neuromuscular sensors arranged circumferentially around a band (e.g., an elastic band) configured to be worn around a user's lower arm (e.g., around a user's forearm). Fig. 7A An embodiment of a wearable system is shown in which neuromuscular sensors 704 (e.g., EMG sensors) are arranged circumferentially around an elastic band 702. It should be understood that any suitable number of neuromuscular sensors may be used, and the number and arrangement of the neuromuscular sensors used may depend on the specific application for which the wearable system is used. For example, a wearable armband or wristband may be used to generate control information to control a robot, control a vehicle, scroll through text, control an avatar, or any other suitable control task. In some embodiments, the elastic band 702 may also include one or more IMUs (not shown).

[0127] For example, Figure 7BAs shown, user 706 can wear elastic band 702 on his / her hand 708. In this way, neuromuscular sensor 704 (e.g., EMG sensor) can be configured to sense and record neuromuscular signals when user 706 uses his / her finger 740 to control or manipulate keyboard 730. In some embodiments, elastic band 702 can also include one or more IMUs (not shown) configured to sense and obtain or record motion information, as described above.

[0128] Figure 8A-8B and Figure 9A-9B Other embodiments of the wearable system of the present technology are shown. Specifically, Fig. 8A A wearable system with multiple sensors 810 is shown, and the sensors 810 are arranged circumferentially around an elastic band 820, which is configured to be worn around the user's lower arm or wrist. The sensors 810 can be neuromuscular sensors (e.g., EMG sensors). As shown, there can be sixteen sensors 810 arranged circumferentially around the elastic band 820 at regular intervals. It should be understood that any suitable number of sensors 810 can be used, and the intervals do not have to be regular. The number and arrangement of sensors 810 can depend on the specific application in which the wearable system is used. For example, when the wearable system is to be worn on the wrist compared to the thigh, the number and arrangement of sensors 810 can be different. Wearable systems (such as armbands, wristbands, thigh bands, etc.) can be used to generate control information for controlling robots, controlling vehicles, scrolling through text, controlling virtual avatars, and / or performing any other suitable control tasks.

[0129] In some embodiments, sensor 810 may include only a set of neuromuscular sensors (e.g., EMG sensors). In other embodiments, sensor 810 may include a set of neuromuscular sensors and at least one auxiliary device. The auxiliary device may be configured to continuously sense and record one or more auxiliary signals. Examples of auxiliary devices include, but are not limited to, IMUs, microphones, imaging devices (e.g., cameras), radiation-based sensors used with radiation-generating devices (e.g., laser scanning devices), heart rate monitors, and other types of devices that can capture a user's condition or other characteristics of the user. Fig. 8A As shown, sensors 810 may be coupled together using flexible electronics 830 incorporated into a wearable system. Figure 8B Shows Fig. 8A A cross-sectional view of one of the sensors 810 of the wearable system is shown.

[0130] In some embodiments, the output of one or more sensing components of sensor 810 may optionally be processed using hardware signal processing circuits (e.g., to perform amplification, filtering, and / or rectification). In other embodiments, at least some of the signal processing of the output of the sensing components may be performed using software. Thus, the signal processing of the signals sampled by sensor 810 may be performed by hardware or software, or by any suitable combination of hardware and software, as aspects of the technology described herein are not limited in this respect. Fig. 9A and Fig. 9B Non-limiting examples of signal processing procedures for processing recorded data from sensor 810 are discussed in more detail.

[0131] Fig. 9A and Fig. 9B Schematic diagram of the internal components of a wearable system with sixteen sensors (e.g., EMG sensors) according to some embodiments of the technology described herein. As shown, the wearable system includes a wearable portion 910 ( Fig. 9A ) and dongle part 920 ( Fig. 9B Although not shown, the dongle portion 920 communicates with the wearable portion 910 (eg, via Bluetooth or another suitable short-range wireless communication technology). Fig. 9A As shown, the wearable portion 910 includes a sensor 810, examples of which are described above in conjunction with Fig. 8A and Figure 8B The sensor 810 provides an output (e.g., a signal) to the analog front end 930, which performs analog processing (e.g., noise reduction, filtering, etc.) on the signal. The processed analog signal generated by the analog front end 930 is then provided to the analog-to-digital converter 932, which converts the processed analog signal into a digital signal that can be processed by one or more computer processors. An example of a computer processor that can be used according to some embodiments is a microcontroller (MCU) 934. Fig. 9A As shown, MCU 934 may also receive inputs from other sensors (e.g., IMU 940) and from power and battery module 942. It should be understood that MCU 934 may receive data from other devices not specifically shown. The processed output of MCU 934 may be provided to antenna 950 for transmission to dongle portion 920, such as Fig. 9B shown.

[0132] The dongle portion 920 includes an antenna 952 that communicates with the antenna 950 of the wearable portion 910. Communication between the antennas 950 and 952 may be performed using any suitable wireless technology and protocol, non-limiting examples of which include radio frequency signaling and Bluetooth. As shown, signals received by the antenna 952 of the dongle portion 920 may be provided to a host computer for further processing, for display, and / or for implementing control of one or more specific physical or virtual objects (e.g., performing control operations in an AR environment).

[0133] Although the reference Fig. 8A , Figure 8B , Fig. 9A and Fig. 9B The examples provided are discussed in the context of an interface with an EMG sensor, but it should be understood that the wearable system described herein may also be implemented with other types of sensors, including but not limited to myograph (MMG) sensors, sonomygraph (SMG) sensors, and electrical impedance tomography (EIT) sensors.

[0134] Back to Figure 1 In some embodiments, the sensor data or signals obtained by the sensor 110 may optionally be processed to compute additional derived measurements, which may then be provided as inputs to an inference model, as described in more detail below. For example, a signal obtained from an IMU may be processed to derive an orientation signal that specifies the orientation of a segment of a rigid body over time. The sensor 110 may implement the signal processing using components that are integrated with the sensing components of the sensor 110, or at least a portion of the signal processing may be performed by one or more components that communicate with but are not directly integrated with the sensing components of the sensor 110.

[0135] The system 100 also includes one or more computer processors 112 programmed to communicate with the sensor 110. For example, signals obtained by the one or more sensors 110 can be output from the sensor 110 and provided to the processor 112, which can be programmed to execute one or more machine learning algorithms to process the signals output by the sensor 110. The algorithm can process the signals to train (or retrain) one or more reasoning models 114, and the trained (or retrained) reasoning models 114 can be stored for later use in generating selection signals and / or control signals for controlling the AR system, as described in more detail below. It should be understood that in some embodiments, the reasoning model 114 may include at least one statistical model.

[0136] In some embodiments, the reasoning model 114 may include a neural network, and may be, for example, a recurrent neural network. In some embodiments, the recurrent neural network may be a long short-term memory (LSTM) neural network. However, it should be understood that the recurrent neural network is not limited to the LSTM neural network, and may have any other suitable architecture. For example, in some embodiments, the recurrent neural network may be any one or any combination of the following: a fully recurrent neural network, a gated recurrent neural network, a recursive neural network, a Hopfield neural network, an associative memory neural network, an Elman neural network, a Jordan neural network, an echo state neural network, a second order recurrent neural network, and / or any other suitable type of recurrent neural network. In other embodiments, a neural network that is not a recurrent neural network may be used. For example, a deep neural network, a convolutional neural network, and / or a feedforward neural network may be used.

[0137] In some embodiments, the inference model 114 can generate discrete outputs. For example, when the desired output is to know whether a specific activation pattern (including individual neural spike events) is detected in the neuromuscular signal, a discrete output (e.g., a discrete classification) can be used. For example, the inference model 114 can be trained to estimate whether the user is activating a specific motor unit, activating a specific motor unit with a specific timing, activating a specific motor unit with a specific discharge pattern, or activating a specific combination of motor units. On a shorter time scale, discrete classification can be used in some embodiments to estimate whether a specific motor unit has discharged an action potential within a given amount of time. In this scenario, these estimates can then be accumulated to obtain an estimated discharge rate for the motor unit.

[0138] In an embodiment where the inference model is implemented as a neural network configured to output a discrete output (e.g., a discrete signal), the neural network may include an output layer that is a softmax layer such that the output of the inference model sums to 1 and can be interpreted as a probability. For example, the output of the softmax layer may be a set of values ​​corresponding to a corresponding set of control signals, where each value indicates a probability that a user wants to perform a particular control action. As a non-limiting example, the output of the softmax layer may be a set of three probabilities (e.g., 0.92, 0.05, and 0.03) indicating a corresponding probability that the detected activity pattern is one of three known patterns.

[0139] It should be understood that when the inference model is a neural network configured to output discrete outputs (e.g., discrete signals), the neural network is not required to produce outputs that sum to 1. For example, instead of a softmax transfer function layer, the output layer of the neural network can be a sigmoid layer that does not restrict the outputs to probabilities that sum to 1. In such an embodiment, the neural network can be trained with a sigmoid cross-entropy cost. This implementation may be advantageous in situations where multiple different control actions may occur within a threshold amount of time and it is not important to distinguish the order in which these control actions occur (e.g., a user may activate two neural activity patterns within a threshold amount of time). In some embodiments, any other suitable non-probabilistic multi-class classifier may be used, as aspects of the technology described herein are not limited in this regard.

[0140] In some embodiments, the output of the inference model 114 may be a continuous signal rather than a discrete output (e.g., a discrete signal). For example, the model 114 may output an estimate of the firing rate of each motor unit, or the model 114 may output a time series of electrical signals corresponding to each motor unit or underlying muscle structure.

[0141] It should be understood that aspects of the technology described herein are not limited to the use of neural networks, as other types of inference models may be employed in some embodiments. For example, in some embodiments, the inference model 114 may include a hidden Markov model (HMM), a switching HMM (where switching allows jumping between different dynamic systems), a dynamic Bayesian network, and / or any other suitable graphical model with a time component. Any such inference model may be trained using sensor signals obtained by the sensor 110.

[0142] As another example, in some embodiments, the inference model 114 may be or may include a classifier that takes as input features derived from a sensor signal obtained by the sensor 110. In such an embodiment, the classifier may be trained using features extracted from the sensor signal. The classifier may be, for example, a support vector machine, a Gaussian mixture model, a regression-based classifier, a decision tree classifier, a Bayesian classifier, and / or any other suitable classifier, as aspects of the technology described herein are not limited in this regard. The input features to be provided to the classifier may be derived from the sensor signal in any suitable manner. For example, the sensor signal may be analyzed as time series data using wavelet analysis techniques (e.g., continuous wavelet transform, discrete time wavelet transform, etc.), Fourier analysis techniques (e.g., short-time Fourier transform, Fourier transform, etc.), and / or any other suitable type of time-frequency analysis techniques. As a non-limiting example, the sensor signal may be transformed using a wavelet transform, and the resulting wavelet coefficients may be provided as input to the classifier.

[0143] In some embodiments, the values ​​of the parameters of the inference model 114 can be estimated based on the training data. For example, when the inference model 114 is or includes a neural network, the parameters (e.g., weights) of the neural network can be estimated based on the training data. In some embodiments, the parameters of the inference model 114 can be estimated using gradient descent, stochastic gradient descent, and / or any other suitable iterative optimization technique. In embodiments where the inference model 114 is or includes a recurrent neural network (e.g., LSTM), the inference model 114 can be trained using stochastic gradient descent and time backpropagation. The training can employ a cross-entropy loss function and / or any other suitable loss function, as aspects of the technology described herein are not limited in this regard.

[0144] The system 100 may also include one or more controllers 116. For example, the controller 116 may include a display controller configured to display a visual representation (e.g., a representation of a hand) on a display device (e.g., a display monitor). As discussed in more detail below, the one or more computer processors 112 may implement one or more trained inference models that receive as input sensor signals obtained by the sensors 110 and provide as output information (e.g., predicted hand state information) for generating control signals that may be used to control, for example, an AR system.

[0145] The system 100 may also optionally include a user interface 118. Feedback determined based on the signals obtained by the sensor 110 and processed by the processor 112 may be provided via the user interface 118 to help the user understand how the system 100 interprets the user's muscle activity (e.g., intended muscle movement). The user interface 118 may be implemented in any suitable manner, including but not limited to an audio interface, a video interface, a tactile interface, and an electrical stimulation interface, or any combination of the foregoing.

[0146] The system 100 may have an architecture that may take any suitable form. Some embodiments may employ a thin architecture in which the processor 112 is or is included as part of a device that is separate from and in communication with the sensor 110 disposed on one or more wearable devices. The sensor 110 may be configured to wirelessly stream sensor signals and / or information derived from the sensor signals to the processor 112 for processing in substantially real time. The device that is separate from and in communication with the sensor 110 may be, for example, any one or any combination of a remote server, a desktop computer, a laptop computer, a smartphone, a wearable electronic device such as a smart watch, a health monitoring device, smart glasses, and an AR system.

[0147] Some embodiments employ a thick architecture, wherein the processor 112 may be integrated with one or more wearable devices on which the sensor 110 is disposed. In yet another embodiment, the processing of the signal obtained by the sensor 110 may be divided between a plurality of processors, at least one of which may be integrated with the sensor 110, and at least one of which may be included as part of a device separate from and in communication with the sensor 110. In such an embodiment, the sensor 110 may be configured to transmit at least some of the sensed signals to a first computer processor remote from the sensor 110. The first computer processor may be programmed to train at least one of the at least one inference model 114 based on the transmitted signal obtained by the sensor 110. The first computer processor may then be programmed to transmit the trained at least one inference model to a second computer processor integrated with one or more wearable devices on which the sensor 110 is disposed. The second computer processor may be programmed to use the trained at least one inference model transmitted from the first computer processor to determine information about the interaction between a user wearing the one or more wearable devices and a physical object in the AR environment. In this way, the training / fitting process and the real-time process utilizing the trained at least one model may be performed separately by using different processors.

[0148] In some embodiments, a computer application simulating an XR environment (e.g., a VR environment, an AR environment, etc.) may be instructed to provide a visual representation by displaying a visual character such as an avatar (e.g., via controller 116). Positioning, movement, and / or forces exerted by portions of the visual character within the XR environment may be displayed based on the output of the trained reasoning model. The visual representation may be dynamically updated as continuous signals are obtained by sensor 110 and processed by trained reasoning model 114 to provide a computer-generated representation of the character's movement that is updated in real time.

[0149] Information obtained by or provided to the system 100 (e.g., input obtained from an AR camera, input obtained from sensor 110) can be used to improve user experience, accuracy, feedback, reasoning models, calibration functions, and other aspects of the overall system. To this end, for example, in an AR environment, the system 100 may include an AR system that includes one or more processors, a camera, and a display that provides AR information within a user's view (e.g., a user interface 118, or other interface via AR glasses or another viewing device). The system 100 may also include system elements that couple the AR system to a computer-based system that generates a musculoskeletal representation based on sensor data (e.g., a signal from at least one neuromuscular sensor). For example, these systems may be coupled via a dedicated or other type of computer system that receives input from an AR system that generates a computer-based musculoskeletal representation. Such systems may include gaming systems, robotic control systems, personal computers, or other systems capable of interpreting AR and musculoskeletal information. The AR system and the system that generates a computer-based musculoskeletal representation may also be programmed to communicate directly. Such information may be transmitted using any number of interfaces, protocols, and / or media.

[0150] As described above, some embodiments involve using an inference model 114 for predicting musculoskeletal information based on signals obtained by wearable sensors. As briefly discussed above in an example in which portions of the human musculoskeletal system may be modeled as a multi-segment articulated rigid body system, the types of joints between segments in the multi-segment articulated rigid body model may be used as constraints to constrain rigid body motion. In addition, different human individuals may move in unique ways when performing tasks, which may be captured in statistical patterns that are generally applicable to individual user behavior. According to some embodiments, at least some of these constraints on human motion may be explicitly incorporated into one or more inference models (e.g., model 114) for predicting user motion. Additionally or alternatively, the constraints may be learned by the inference model 114 through training based on sensor data obtained from the sensor 110, as briefly discussed above.

[0151] Some embodiments relate to using an inference model for predicting information to generate a computer-based musculoskeletal representation and / or update a computer-based musculoskeletal representation in real time. For example, the predicted information may be predicted hand state information. The inference model may be used to predict information based on an IMU signal, a neuromuscular signal (e.g., an EMG, MMG, and / or SMG signal), an external or auxiliary device signal (e.g., a camera or laser scanning signal), or a combination of an IMU signal, a neuromuscular signal, and an external or auxiliary device signal detected when the user performs one or more movements. For example, as described above, a camera associated with an AR system may be used to capture data of the actual positioning of a human object based on a computer-based musculoskeletal representation, and such actual positioning information may be used to improve the accuracy of the representation. In addition, the output of the inference model may be used to generate a visual representation of a computer-based musculoskeletal representation in an AR environment. For example, a visual representation of muscle group discharge, force application, text input through movement, or other information generated by a computer-based musculoskeletal representation may be rendered in a visual display of an AR system. In some embodiments, other input / output devices (e.g., an auditory input / output terminal, a tactile device, etc.) may be used to further improve the accuracy of the entire system and / or improve the user experience.

[0152] Some embodiments of the technology described herein involve using an inference model to at least partially map muscle activation state information to control signals, wherein the muscle activation state information is information identified from neuromuscular signals obtained by a neuromuscular sensor. When a user performs one or more muscle activations, one or more movements, and / or one or more gestures, the inference model can receive as input detected IMU signals, neuromuscular signals (e.g., EMG, MMG, and SMG signals), external device signals (e.g., camera or laser scanning signals), or a combination of IMU signals, neuromuscular signals, and external or auxiliary device signals. The inference model can be used to predict control information without requiring the user to make perceptible movements.

[0153] As described above, according to some embodiments of the present technology, camera information can be used to improve the interpretation of neuromuscular signals and their relationship to movement, positioning, and force generation. It should be understood that camera information can be, for example, an image signal corresponding to at least one image captured by a camera; therefore, as used herein, an image from a camera can be understood to refer to an image signal from a camera. The camera can be a still camera, a video camera, an infrared camera, etc. that is capable of capturing or recording an image of a user. One or more filters can be used on the camera so that the camera can capture images only within a specific range of light wavelengths. It should be understood that the image can be a still image, a still image sequence (or image sequence), a moving image (or video sequence), etc., which can be captured and recorded as a signal. The terms "camera information", "camera data", and "camera signal" can be used herein to refer to information about a user that can be captured by a camera. It should be understood that although various embodiments may refer to "one" camera or "the" camera, these embodiments may utilize two or more cameras instead of one camera. In addition, the camera information may relate to any one or any combination of the following: images produced by visible light, images produced by invisible (e.g., infrared) light, images produced by light of a predetermined wavelength range, and images produced by light of two or more different predetermined wavelength ranges. For example, invisible light may be used to capture images showing heat distribution within a user, which may provide an indication of blood flow within the user, which in turn may be used to infer the user's condition (e.g., a user's finger applying force may have a different blood flow pattern than a finger not applying force).

[0154] The camera may be mounted on the user (e.g., mounted on a head mounted display worn by the user, or mounted on a glove worn on the user's hand), or may be mounted external to the user to capture the user and / or the user's environment. When the camera is mounted on the user, the camera may be used to capture the user's environment and / or portions of the user's body (e.g., a handheld camera may be used to capture an image of the user's other hand).

[0155] Fig.10 1 is a diagram illustrating an example implementation of a system 1000 utilizing one or more EMG sensors 1040 and a camera 1060 according to some embodiments of the techniques described herein. For example, Fig.10A user's arm and attached hand ("arm / hand") 1010 is shown, which is composed of one or more joints and segments and can be depicted as a musculoskeletal representation. More specifically, the user's hand segments 1020 are connected by joints. The positioning of the arm and hand and the segment lengths of the arm and hand can be determined by the system 1000 and positioned within the three-dimensional space of the model musculoskeletal representation. In addition, the user's hand can also include an interpolated forearm segment 1030. As described above, a neuromuscular activity system can be used to determine one or more representations of the user's hand / arm positioning. To this end, the user can wear a band including one or more EMG sensors 1040 that sense and record neuromuscular signals used to determine the musculoskeletal representation. While the EMG sensors 1040 are sensing and recording the neuromuscular signals, a camera 1060 can be used to capture objects within the camera's field of view 1050. For example, in Fig.10 , the camera's field of view 1050 includes the user's arm / hand 1010. In addition to the neuromuscular activity signals determined by the EMG sensor 1040, the camera data can also be used to reconstruct the positioning, geometry, and forces applied by the user's arm / hand 1010. In addition, an output from the system 1000 can be provided that allows the system 1000 to render a representation of the user's arm / hand 1010, such as in an AR environment of an AR system.

[0156] Figure 2 FIG. 2 shows a schematic diagram of an AR-based system 200, which may be a distributed computer-based system that integrates an AR system 201 with a neuromuscular activity system 202. The neuromuscular activity system 202 is similar to the one described above with reference to FIG. Figure 1 System 100 is described.

[0157] Typically, an XR system such as an AR system 201 may take the form of a pair of goggles or glasses or eye protection, or other type of display device that displays display elements to a user that may be superimposed on the user's "reality." In some cases, the reality may be the user's observation of the environment (e.g., as viewed through the user's eyes), or a captured version of the user's observation of the environment (e.g., via a camera). In some embodiments, the AR system 201 may include one or more cameras 204, which may be mounted within a device worn by the user, capturing one or more observations experienced by the user in the user's environment. The system 201 may have one or more processors 205 that operate within the device worn by the user and / or within a peripheral device or computer system, and such processors 205 may be capable of sending and receiving video information and other types of data (e.g., sensor data).

[0158] The AR system 201 may also include one or more sensors 207, such as a microphone, a GPS element, an accelerometer, an infrared detector, a tactile feedback element, or any other type of sensor, or any combination thereof. In some embodiments, the AR system 201 may be an audio-based or auditory AR system, and the one or more sensors 207 may also include one or more headphones or speakers. In addition, the AR system 201 may also have one or more displays 208, which allow the AR system 201 to overlay and / or display information to the user in addition to providing the user with a view of the user environment presented via the AR system 201. The AR system 201 may also include one or more communication interfaces 206, which enable information to be transmitted to one or more computer systems (e.g., a gaming system or other system capable of rendering or receiving AR data). AR systems can take many forms and can be obtained from many different manufacturers. For example, various embodiments may be implemented in association with one or more types of AR systems or platforms, such as the R-7 holographic reality glasses available from Microsoft Corporation (Redmond, Washington, USA); the Lightwear AR head-mounted device from Magic Leap (Plantation, Florida, USA); the Google Glass AR glasses available from Alphabet (Mountain View, California, USA); the R-7 smart glasses system available from Osterhout Design Group (also known as ODG, San Francisco, California, USA); the Oculus Quest, Oculus Rift S, and SparkAR Studio available from Facebook (Menlo Park, California, USA); or any other type of AR or other XR device. Although discussed using AR by way of example, it should be understood that one or more embodiments of the technology disclosed herein may be implemented in one or more XR systems.

[0159] The AR system 201 can be operably coupled to the neuromuscular activity system 202 via one or more communication schemes or methods, including but not limited to Bluetooth protocol, Wi-Fi, Ethernet-like protocol, or any number of wireless and / or wired connection types. It should be understood that, for example, the systems 201 and 202 can be directly connected or coupled through one or more intermediate computer systems or network elements. Figure 2 The bidirectional arrow in represents the communication coupling between systems 201 and 202 .

[0160] As described above, the neuromuscular activation system 202 may be similar in structure and function to the Figure 1The system 100 described herein. In particular, the system 202 may include one or more neuromuscular sensors 209, one or more inference models 210, and may create, maintain, and store a musculoskeletal representation 211. In an exemplary embodiment, similar to the embodiments discussed above, the system 202 may include or may be implemented as a wearable device, such as a band that can be worn by a user, so as to collect (i.e., obtain) and analyze neuromuscular signals from the user. In addition, the system 202 may include one or more communication interfaces 212 that allow the system 202 to communicate with the AR system 201, such as via Bluetooth, Wi-Fi, or other communication methods. Notably, the AR system 201 and the neuromuscular activity system 202 may communicate information that can be used to enhance the user experience and / or allow the AR system 201 to operate more accurately and efficiently.

[0161] although Figure 2 A computer-based distributed system 200 is shown that integrates an AR system 201 with a neuromuscular activity system 202, but it should be understood that the integration of these systems 201 and 202 can be non-distributed in nature. In some embodiments, the neuromuscular activity system 202 can be integrated into the AR system 201, so that various components of the neuromuscular activity system 202 can be considered as part of the AR system 201. For example, input from neuromuscular signals recorded by the neuromuscular sensor 209 can be considered as another input to the AR system 201 (e.g., from the camera 204, from the sensor 207). In addition, the processing of the input obtained from the neuromuscular sensor 209 (e.g., the obtained sensor signal) can be integrated into the AR system 201.

[0162] Figure 3 A process 300 is shown for providing a user with enhanced interaction with physical objects in an AR environment generated by an AR system (such as AR system 201) using neuromuscular signals, according to some embodiments of the techniques described herein. Process 300 may be performed at least in part by neuromuscular activity system 202 and / or AR system 201 of AR-based system 200. In action 310, sensor signals (also referred to herein as "raw sensor signals") may be obtained (e.g., sensed and recorded) by one or more sensors of neuromuscular activity system 202. In some embodiments, the sensors may include a plurality of neuromuscular sensors 209 (e.g., EMG sensors) disposed on a wearable device worn by a user. For example, sensor 209 may be an EMG sensor disposed on an elastic band configured to be worn around a wrist or forearm of a user to sense and record neuromuscular signals from the user when the user performs muscle activation (e.g., movement, gesture). In some embodiments, the EMG sensor may be sensor 704 disposed on band 702, such as Fig. 7A In some embodiments, the EMG sensor may be a sensor 810 disposed on a belt 820, such as Fig. 8A The muscle activations performed by the user can include static gestures, such as placing the user's palm on a table; dynamic gestures, such as waving fingers back and forth; and covert gestures that are not noticeable to others, such as slightly tensing a joint by co-contracting opposing muscles or using submuscular activation. The muscle activations performed by the user can include symbolic gestures (e.g., gestures that are mapped to other gestures, interactions, or commands based on a gesture vocabulary that specifies mappings).

[0163] In addition to the plurality of neuromuscular sensors 209, in some embodiments of the technology described herein, the neuromuscular activity system 202 may include one or more auxiliary sensors configured to obtain (e.g., sense and record) auxiliary signals, which may also be provided as input to one or more trained inference models, as described above. Examples of auxiliary sensors include an IMU, an imaging device, a radiation detection device (e.g., a laser scanning device), a heart rate monitor, or any other type of biosensor capable of sensing biophysical information from a user during the execution of one or more muscle activations. In addition, it should be understood that some embodiments of the present technology may be implemented using a camera-based system that performs skeletal tracking, such as the Kinect system available from Microsoft Corporation (Redmond, Washington, USA) and the Leap Motion system available from Leap Motion, Inc. (San Francisco, California, USA). It should be understood that the various embodiments described herein may be implemented using any combination of hardware and / or software.

[0164] Process 300 then proceeds to action 320, where raw sensor signals, which may include signals sensed and recorded by one or more sensors (e.g., EMG sensors, auxiliary sensors, etc.) and optional camera input signals from one or more cameras, may be optionally processed. In some embodiments, raw sensor signals may be processed using hardware signal processing circuits (e.g., performing amplification, filtering, and / or rectification). In other embodiments, at least some signal processing of raw sensor signals may be performed using software. Thus, signal processing of raw sensor signals sensed and recorded by one or more sensors and optionally obtained from one or more cameras may be performed using hardware, software, or any suitable combination of hardware and software. In some embodiments, raw sensor signals may be processed to derive other signal data. For example, accelerometer data obtained by one or more IMUs may be integrated and / or filtered to determine derived signal data associated with one or more muscles during muscle activation or gesture execution.

[0165] Process 300 then proceeds to action 330, where one or more visual indicators may be optionally displayed in the AR environment based at least in part on the neuromuscular signals obtained by the plurality of neuromuscular sensors 209. For example, the AR system 201 may operate in conjunction with the neuromuscular activity system 202 to overlay one or more visual indicators on or near a physical object in the AR environment. The one or more visual indicators may indicate to the user that the physical object is an object having a set of virtual controls associated therewith, such that if the user interacts with the object (e.g., by picking it up), the user may perform one or more "enhanced" or "enhanced" interactions with the object. The one or more visual indicators may be displayed in the AR environment in any suitable manner. For example, the physical object may change color or glow, thereby indicating that it is an object capable of enhanced interaction. In another example, an indication of a set of virtual controls of a physical object that can be activated by the user to control the physical object may be overlaid on or displayed near the physical object in the AR environment. The user may interact with an indicator of a set of virtual controls by, for example, performing muscle activation to select one of the virtual controls. In response to the user's interaction with the indicator of the set of virtual controls, information about the interaction with the physical object can be determined. For example, if the physical object is a writing instrument and the displayed indicator of the set of virtual controls indicates that the user can use the writing instrument as a pen, a brush, or a pointing device in the AR environment, the user can perform a gesture to select a brush function so that when the user picks up the writing instrument, it can be used to draw in the AR environment.

[0166] Process 300 then proceeds to act 340, where information about the user's interaction with the physical object is determined based, at least in part, on the neuromuscular signals obtained by the plurality of neuromuscular sensors 209 and / or information derived from the neuromuscular signals. Optionally, auxiliary signals from one or more auxiliary devices (e.g., a camera, an IMU, etc.) may supplement the neuromuscular signals to determine information related to the user's interaction with the physical object. For example, based, at least in part, on the neuromuscular signals (and optionally supplemented with the auxiliary signals), the AR-based system 200 may determine how tightly the user is gripping the physical object, and may send a control signal to the AR-based system 200 based on the amount of gripping force applied to the physical object. Continuing with the example above, the physical object may be a writing instrument, and applying different amounts of gripping force to the surface of the writing instrument and / or pressing different portions of the writing instrument may transform the writing instrument into an "enhanced" or "augmented" writing instrument in which a set of virtual control actions for the physical object may be enabled or activated.

[0167] In some embodiments, information about a user's interaction with a physical object in an AR environment may be determined based on a combination of neuromuscular signals and at least one other sensor (e.g., a camera, an IMU, etc.). For example, some embodiments may include at least one camera (e.g., as part of AR system 201), which may be arranged or configured to capture one or more images. Fig.10 An example of such an arrangement is shown. Neuromuscular signals obtained by multiple neuromuscular sensors 209 and images captured by the camera can be used, for example, to determine the force applied by the user to the physical object. Neuromuscular signal data and auxiliary sensor data (e.g., camera data) can be combined in any other suitable manner to determine information associated with the user's interaction with the physical object, and the embodiments are not limited in this respect.

[0168] Process 300 then proceeds to action 350, where feedback based on the determined information about the user's interaction with the physical object is provided. In some embodiments, feedback is provided to the user interacting with the object. For example, the AR-based system 200 can provide feedback (e.g., visual feedback, auditory feedback, tactile feedback) to the user in the AR environment. In embodiments where visual feedback is provided in the AR environment, the visual feedback can be provided in any suitable manner. For example, the physical object with which the user interacts can change color or glow, indicating that the user is interacting with the object. Alternatively, feedback can be provided using a visual indicator separate from the physical object. For example, an icon or other visual indicator can be displayed in the AR environment, showing the interaction mode (e.g., brush mode) of the object with which the user is interacting. In some embodiments where feedback is provided to the user, feedback can be provided using non-visual forms of feedback such as auditory or tactile feedback. The feedback can, for example, indicate to the user that the physical object with which he / she is interacting may have enhanced properties or functions that may not be available through ordinary real-world interactions with the object.

[0169] In some embodiments of the technology described herein, an AR system (e.g., system 201) may include a tactile circuit capable of transmitting tactile signals to a user. The tactile signal can be used to provide feedback to the user and may include any one or any combination of a vibration actuator, a skin tap actuator, a low-voltage electric shock circuit, and a force actuator. Such tactile actuators are known in the art and may include electromagnetic transducers, motors, etc. The tactile circuit can be arranged on a wearable device worn by the user and may be included on a wearable patch or a wearable band, such as those discussed above. For example, a tactile circuit may be included on band 702 together with one or more neuromuscular sensors.

[0170] The AR system can be controlled to provide feedback to the user as tactile feedback delivered via a tactile circuit. In some embodiments, the AR system can be controlled in this regard by a controller within the AR system or by a controller of an AR-based system (e.g., system 200) that includes the AR system. In other embodiments, the AR system can be controlled in this regard by control signals from a controller external to the AR system and external to the AR-based system.

[0171] In some embodiments, feedback may be provided to the user as a changed functionality of the physical object itself, rather than as an indication separate from the changed functionality. For example, feedback may be provided to the user based on the current functionality of one or more physical objects. Using the example above, a user may pick up a writing tool in an AR environment and hold the writing tool with a certain amount of force. Grasping the writing tool with a certain amount of force may convert the physical writing tool into an enhanced writing tool (or AR writing tool), which may have different writing characteristics for writing in an AR environment. When the user writes in an AR environment using an enhanced writing tool, the enhanced functionality of the writing tool may become apparent by using the physical object itself to write. For example, the color of the writing produced by the enhanced writing tool, and / or the pixel size of the writing produced by the enhanced writing tool, and / or some other writing characteristics or a combination of writing characteristics may provide feedback on the enhanced functionality of the physical object.

[0172] In some embodiments, feedback can be provided to others besides the user interacting with the physical object. For example, in a shared AR environment that includes a first user interacting with the object and a second user not interacting with the object, an indication of the interaction between the first user and the physical object in the AR environment can be provided to the second user. This feedback can enable the second user to understand how the first user interacts with the physical object and one or more other physical objects in the AR environment without directly observing the first user interacting with the object. For example, the second user is able to determine how hard the first user is gripping a ball in the AR environment, which may not be visually apparent by observing the first user interacting with the ball.

[0173] The feedback provided in action 350 can reflect information about the user's interaction with the physical object in any suitable manner. For example, in some embodiments, the information about the interaction can include force information related to the force that the user is applying to the object (e.g., by pressing, grasping, etc.). In such embodiments, a visual rendering of the force can be displayed in the AR environment to allow the user (or another user in a shared AR environment) to visualize the amount of force applied to the object. In other embodiments where force information is included in the information about the interaction, non-visual techniques can be used to provide feedback. For example, auditory and / or tactile feedback can be provided based at least in part on the amount of force applied to the object.

[0174] As described above, in some embodiments of the technology disclosed herein, physical objects in an AR environment may be converted into "augmented" objects. An augmented object may have a set of enhanced or enhanced features in an AR environment, and such enhanced features may not be available when a user interacts with the object in a real environment. For example, a writing instrument (such as a pen) can typically only write with a single color of ink provided in the pen and use a line width determined in part by the nib. In some embodiments, such a pen can be converted into an augmented pen that is endowed with a set of enhanced features for use in an AR environment. For example, when used in an AR environment, an augmented writing instrument may have a set of enhanced features that enable selection from a variety of writing colors, line thicknesses, brush shapes, nib shapes, drawing modes (e.g., a pen up / down function so that writing occurs only when a certain amount of pressure is applied to the writing instrument), and writing instrument types (e.g., a paintbrush, a spray can, a pen, a pencil, a highlighter). In some embodiments, the enhanced features may also include functions that are not typically associated with the use of objects in a real-world environment. For example, an enhanced writing tool can be used as a remote control or pointer in an AR environment to select and / or manipulate one or more other objects at a distance from the user in the AR environment. In another embodiment, physical objects that are not typically used as writing tools (e.g., a stick, a ruler, or other object that can be held in the user's hand) can be converted into writing tools for use in an AR environment, based at least in part on the user's interaction with the physical object. For example, a user can pick up a physical object (e.g., a stick) in their environment and "doublegrasp" the object (i.e., grab the object twice) with a force greater than a threshold amount to convert the object into a writing tool in the AR environment, thereby converting the physical object into an enhanced object that can be used to provide writing input in the AR environment.

[0175] In some embodiments, the user's selection (e.g., selection of a physical object and / or a function associated with a physical object) can be performed at least in part based on user context and / or user behavior. For example, the selection of a physical object can be based at least in part on user behavior, such as information about one or more recent interactions between a user and one or more physical objects. In another example, if the user recently controlled a particular device (such as a smart speaker) by pressing play on a new song, the smart speaker can be automatically selected, and the volume of the selected smart speaker can be changed using muscle activation determined at least in part based on the sensed neuromuscular signals. In another example, the selection of a physical object can be based at least in part on user context, such as information about the user's current location (e.g., which environment (e.g., room) the user is currently located in). Information about the user's current location can be determined in any suitable manner, including but not limited to using NFC technology, RFID technology, another field-based technology, and global positioning technology (e.g., GPS). Based at least in part on location information, a physical object in the user's environment (e.g., a light switch in a particular room) can be selected for control.

[0176] Figure 4 A process 400 for enabling and / or disabling virtual controls associated with objects in an AR environment is shown in accordance with some embodiments of the technology disclosed herein. At act 410, a plurality of neuromuscular signals are obtained by a plurality of neuromuscular sensors (e.g., neuromuscular sensors 209) worn by a user. For example, the plurality of neuromuscular signals may be sensed and recorded by the plurality of neuromuscular sensors. Process 400 then proceeds to act 420, where the plurality of neuromuscular signals may optionally be processed (e.g., amplified, filtered, rectified, etc.), examples of which are discussed above. Process 400 then proceeds to act 430, where the plurality of neuromuscular signals and / or information derived from the plurality of neuromuscular signals are interpreted, for example, to determine muscle activation performed by the user. Examples of interpreting neuromuscular signals may include, but are not limited to, processing multiple neuromuscular signals using one or more trained inference models (e.g., inference model 114) to identify muscle activation states that a user has performed, determining an amount of force applied to an object with which the user is interacting (e.g., an amount of force used to hold or grasp an object, an amount of force used to push against a stationary surface such as a table, etc.), determining co-contractile forces, and determining submuscular activation (e.g., activation of a single motor unit).

[0177] Process 400 then proceeds to act 440, where it is determined whether a set of virtual controls will be enabled (if currently disabled) or disabled (if currently enabled) based at least in part on the interpreted neuromuscular signals. Whether to enable or disable a set of virtual controls may be determined in any suitable manner. Some embodiments map specific muscle activation states (e.g., specific gestures, or specific muscle tension, or specific muscle activation, or any combination thereof) to control signals for enabling or disabling a set of virtual controls associated with a physical object. For example, in some embodiments, a neuromuscular activity system (e.g., neuromuscular activity system 202) may be configured to associate a pinch gesture involving a user's thumb and index finger with a command to enable or disable a set of virtual controls for a physical object. In other embodiments, the neuromuscular activity system may be configured to associate detection of a specific amount of force applied to an object with which the user is interacting with a command to enable or disable a set of virtual controls for the physical object. The amount of force applied to the object may be an amount of static force applied at a single point in time, or a dynamic sequence of forces applied to the object (e.g., a double squeeze of an object may enable or disable a set of virtual controls). In other embodiments, the neuromuscular activity system can be configured to associate the activation of a single motor unit or a defined set of motor units with a command to enable or disable a set of virtual controls for a physical object. At action 440, the output of the neuromuscular signal interpretation process at action 430 can be compared with stored information associating one or more muscle activation states with control signals for enabling or disabling a set of virtual controls for a physical object to determine whether to enable or disable a set of virtual controls for the object. In some embodiments, the same muscle activation state can be used to enable and disable virtual controls for all physical objects in the AR environment. In other embodiments, different muscle activation states can be associated with enabling and disabling virtual controls for different objects in the AR environment.

[0178] In some embodiments, an AR-based system (e.g., system 200) may be configured to automatically enable virtual controls for physical objects in an AR environment without requiring the user to perform muscle activation. For example, a component of the AR-based system may be configured to determine which physical object(s) near the user has virtual controls associated with the object, and the AR-based system may automatically enable virtual controls for those physical objects, and may optionally provide appropriate feedback to the user and / or one or more other users (e.g., in a shared AR environment). Alternatively, the AR-based system may automatically enable virtual controls for a particular physical object in response to a user's interaction with the object (e.g., when the user touches the object). Regardless of whether the AR-based system automatically enables virtual controls for an object, or requires the user to perform muscle activation to enable virtual controls, the AR-based system may disable virtual controls for an object in response to interpreting neuromuscular signals obtained from the user and / or information derived from the neuromuscular signals. In some embodiments, the muscle activation state for enabling virtual controls is the same as the muscle activation state for disabling virtual controls. In other embodiments, the muscle activation states for enabling and disabling virtual controls for one or more objects in an AR environment are different.

[0179] If it is determined in act 440 that virtual controls for physical objects in the AR environment should be enabled or disabled, the process proceeds to act 450, in which virtual controls for physical objects are enabled or disabled. For example, a control signal may be sent to an AR system (e.g., system 201) instructing the AR system to enable or disable virtual controls for a particular physical object, all physical objects near the user in the AR environment, or all physical objects within the user's field of view in the AR environment. In some embodiments, enabling a set of virtual controls for a physical object includes providing an indication that the physical object has been converted into an augmented object to which the virtual controls apply. Examples of providing indications may include, but are not limited to, providing visual, auditory, and / or tactile indications to the user in the AR environment.

[0180] When virtual controls for an augmented or enhanced physical object are enabled, process 400 may optionally proceed to act 460 where a determination is made as to the user's interaction with the object using the enabled set of virtual controls. For example, if the augmented physical object is a writing instrument and a set of enabled augmented features for the object includes a brush mode, a pointer mode, and a remote control mode, the user may select one of the three modes by performing different muscle activations sensed and interpreted by a neuromuscular activity system (e.g., system 202).

[0181] The AR environment may include multiple physical objects with which a user can interact. One or more of the multiple physical objects in the AR environment may have a set of control actions associated therewith. For example, as described above, an AR system (e.g., system 201) based on an AR system (e.g., system 200) may be configured to associate a set of virtual controls with physical objects in the AR environment, and neuromuscular control signals obtained by a neuromuscular activity system (e.g., system 202) may be used to enable or activate the set of virtual controls. From the multiple physical objects in the AR environment, the AR system may determine which object the user is interacting with, and may activate an appropriate set of control actions for the determined object.

[0182] As described herein, AR components (eg, cameras, sensors, etc.) may be used in conjunction with a neuromuscular controller (eg, one or more controllers 116 ) to provide enhanced functionality for interacting with physical objects in an AR environment. Figure 5A process 500 for activating a set of control actions associated with physical objects according to some embodiments of the technology disclosed herein is shown. In action 510, one or more cameras (e.g., camera 204) associated with an AR-based system (e.g., system 200) are used to identify physical objects in an AR environment. Images captured by the camera may be particularly useful in mapping the environment and identifying physical objects in the AR environment. For example, if a user of an AR-based system is located in the user's kitchen, a camera associated with the AR-based system may detect multiple physical objects in the user's AR environment, such as a refrigerator, a microwave, a stove, a pen, or an electronic device on a kitchen counter. As described above, in some embodiments, at least some of the physical objects in the AR environment may be associated with virtual controls that, when enabled, allow the user to interact with the objects in a way that is not possible when the user interacts with objects in the real world environment. In an exemplary embodiment of the technology disclosed herein, visual feedback may be provided to the user in the AR environment to help guide or direct the user in the real world. For example, a set of dials in the real world may be represented in an AR environment with an overlaying visual interface and / or graphics to inform the user about each dial, such as the type of dial, the range of settings for the dial, the purpose of the dial, or another feature of the dial as it relates to the real world and the user's interaction with the dial in the real world. In another example, visual instructions may be overlaid on piano keys in an AR environment to instruct a user in the real world how to play a piece of music on the piano. In another exemplary embodiment, if a stove in a kitchen has been used recently and the surface of the stove is still hot, the AR system may present a warning label to the user in the AR environment so that the user can avoid being burned by the stove in real life. As another example, the AR system may present status information in an AR environment to provide the user with information about the status of a physical object, such as device battery life, whether the device is turned on or off, etc.

[0183] Process 500 then proceeds to action 512, where the user's interaction with the physical object in the AR environment is identified. The identification of the interaction can be performed based at least in part on one or more images captured by a camera associated with the AR system. For example, it can be determined from one or more images that the user is holding a physical object (e.g., a writing tool), touching an object (e.g., the surface of a table), or reaching out to an object (e.g., a thermostat on a wall). In some embodiments, as described above, the identification of the user's interaction with the object and / or the determination of how the user interacts with the object can be performed based at least in part on multiple neuromuscular signals obtained (e.g., sensed and recorded) by a wearable sensor. For example, it can be determined at least in part based on neuromuscular signals that the user is interacting with the object by pushing the object with a certain force. After identifying the user's interaction with the physical object, process 500 proceeds to action 514, in which a set of control actions associated with the object with which the user is interacting is activated. For example, the set of control actions can be virtual controls, examples of which are described above.

[0184] In some embodiments, the interpretation of neuromuscular signals by at least one trained inference model can be based at least in part on a particular object with which the user is interacting. For example, information about a physical object and / or information associated with a set of activated control actions can be provided as input to a trained inference model for interpreting neuromuscular signals.

[0185] In some embodiments, the operation of the AR-based system can be modified based at least in part on the user's interaction with the physical objects in the AR environment. For example, the mode of the AR-based system can be changed from a coarse interaction mode (e.g., determining whether the user is interacting with a physical object in the AR environment) to a higher precision interaction mode for detecting a finer-grained (e.g., more subtle or more detailed) interaction of the user with the physical object. The finer-grained interaction may include information about how the user interacts with the physical object to perform different tasks in the AR environment using the physical object. When determining information about one or more interactions of the user with one or more physical objects in the AR environment, the higher precision interaction mode of the AR-based system may, for example, weight neuromuscular signals more strongly than input from other sensors (e.g., a camera of the AR-based system).

[0186] In some embodiments, neuromuscular signals may be used, at least in part, to influence how objects in an AR environment are digitized or rendered by an AR system. For some embodiments, an AR application may use one or more cameras to scan a physical environment to create a 3D model of the environment and physical objects within the physical environment, so that, for example, virtual objects may be appropriately placed within an AR environment that may be generated based on the physical environment in a manner that enables the virtual objects to properly interact with physical objects in the AR environment. For example, a virtual character created in the generated AR environment may be rendered as jumping up and down on a physical table in the AR environment. To ensure that the virtual character is correctly represented in the AR environment, properties about the table and its positioning are characterized in the 3D model of the physical environment created by the AR system. A potential limitation of camera-based scanning technology for creating a 3D model of the physical environment is that invisible properties of objects in the environment (including but not limited to weight, texture, compressibility, bulk modulus, center of mass, and elasticity) are inferred from the visible appearance of the objects. In some embodiments, when a user interacts with a physical object in a physical environment (e.g., by picking up, pressing, or otherwise manipulating the object), information about at least one invisible attribute of the object can be more specifically determined based on neuromuscular signals obtained by a wearable sensor worn by the user, and this information about the invisible attribute of the object can be used to create a more accurate model of the object in the AR environment. For example, the neuromuscular signals obtained when a user picks up a soft, plush pillow may be different from the neuromuscular signals obtained when the user picks up a can of cold soda. Differences in, for example, texture, temperature, and / or elasticity of a physical object may not be easily seen, but can be sensed by one or more neuromuscular sensors and / or one or more auxiliary sensors and provided to the user as feedback.

[0187] The above embodiments can be implemented in any of a variety of ways. For example, the embodiments can be implemented using hardware, software, or a combination thereof. When implemented in software, the code constituting the software can be executed on any suitable processor or processor set, whether provided in a single computer or distributed in multiple computers. It should be understood that any component or component cluster that performs the above functions can be generally considered to be one or more controllers that control the functions discussed above. One or more controllers can be implemented in a variety of ways, such as using dedicated hardware or using one or more processors that are programmed to perform the functions listed above using microcode or software.

[0188] In this regard, it should be understood that one implementation of an embodiment of the present invention includes at least one non-transitory computer-readable storage medium (e.g., computer memory, portable storage, optical disk, etc.) encoded with a computer program (i.e., a plurality of instructions), which, when executed on a processor (or multiple processors), performs the above-mentioned functions of the embodiments of the technology described herein. The at least one computer-readable storage medium may be transportable, so that the program stored thereon can be loaded onto any computer resource to implement the aspects of the present invention discussed herein. In addition, it should be understood that the reference to a computer program that performs the functions discussed above when executed is not limited to an application program running on a host computer. Instead, the term "computer program" is used in a general sense herein to refer to any type of computer code (e.g., software or microcode) that can be used to program a processor to implement the above-discussed aspects of the present invention. It should be understood that a first portion of the program may be executed on a first computer processor and a second portion of the program may be executed on a second computer processor that is different from the first computer processor. The first and second computer processors may be located at the same location or at different locations; in each case, the first and second computer processors may communicate with each other via, for example, a communication network.

[0189] In some embodiments of the present technology provided herein, a kit for controlling an AR system may be provided. The kit may include a wearable device including a plurality of neuromuscular sensors configured to sense a plurality of neuromuscular signals of a user, and a non-transitory computer-readable storage medium encoded with a plurality of instructions, which, when executed by at least one computer processor, cause the at least one computer processor to execute a method for enabling a user to interact with a physical object in an AR environment generated by an AR system. The method may include: receiving a plurality of neuromuscular signals sensed from a user by a plurality of neuromuscular sensors as input; determining information about the user's interaction with a physical object in an AR environment generated by the AR system based at least in part on the plurality of neuromuscular signals; and instructing the AR system to provide feedback based at least in part on the information about the user's interaction with the physical object. For example, as described above, the wearable device may include a wearable band configured to be worn around a part of a user, or a wearable patch configured to be worn on a part of a user.

[0190] The various aspects of the present invention may be used alone, in combination, or in various arrangements not specifically discussed in the above embodiments, and therefore their application is not limited to the details and arrangements of the components shown in the foregoing description and / or the accompanying drawings. For example, aspects described in one embodiment may be combined in any manner with aspects described in other embodiments.

[0191] In addition, embodiments of the present invention may be implemented as one or more methods, of which at least one example is provided. The actions performed as part of a method may be ordered in any suitable manner. Thus, embodiments may be constructed in which the actions are performed in an order different from that shown and / or described, which may include performing some actions simultaneously, even though shown and / or described as sequential actions in the illustrative embodiments.

[0192] The use of ordinal terms such as "first," "second," "third," etc. in the claims to modify claim elements does not, by itself, imply any priority, precedence, or order of one claim element relative to another claim element or a temporal order in which the actions of a method are performed. Such terms are merely used as labels to distinguish one claim element having a particular name from another element having the same name (but for the use of ordinal terms).

[0193] The phraseology and terminology used herein is for the purpose of description and should not be regarded as limiting. The use of “including,” “comprising,” “having,” “containing,” “involving,” and variations thereof is meant to include the items listed thereafter and additional items.

[0194] In addition, although the advantages of the technology described herein can be pointed out, it should be understood that not every embodiment of the present invention includes every described advantage. Some embodiments may not implement any feature described herein as advantageous. Therefore, the foregoing description and drawings are only intended as examples.

[0195] Variations of the disclosed embodiments are possible. For example, various aspects of the present technology may be used alone, in combination, or in various arrangements not specifically discussed in the previously described embodiments, and therefore they are not limited in application to the details and arrangements of components set forth in the foregoing description or shown in the accompanying drawings. Aspects described in one embodiment may be combined in any manner with aspects described in other embodiments.

[0196] The use of ordinal terms such as "first," "second," "third," etc. in the claims to modify claim elements does not, by itself, imply any priority, precedence, or order of one claim element relative to another claim element or a temporal order in which the actions of a method are performed, and such terms serve merely as labels to distinguish one claim element having a particular name from another element having the same name (but for the use of ordinal terms).

[0197] The indefinite articles "a" and "an" as used in the specification and claims should be understood to mean "at least one" unless explicitly indicated to the contrary.

[0198] With respect to a list of one or more elements, any use of the phrase "at least one" should be understood to mean at least one element selected from any one or more elements in the list of elements, but not necessarily including at least one of each element specifically listed in the list of elements, and not excluding any combination of elements in the list of elements. This definition also allows that elements other than the elements specifically identified within the list of elements to which the phrase "at least one" refers may optionally be present, whether related or unrelated to those elements specifically identified.

[0199] Any use of the phrase "equal" or "same" in reference to two values ​​(e.g., distance, width, etc.) means that the two values ​​are the same within a manufacturing tolerance. Thus, two values ​​being equal or the same may mean that the two values ​​differ from each other by ±5%.

[0200] The phrase "and / or" as used herein in the specification and in the claims should be understood to mean "either or both" of the elements so connected, i.e., elements that are present connectedly in some cases and separately in other cases. Multiple elements listed with "and / or" should be interpreted in the same manner, i.e., "one or more" of the elements so connected. In addition to the elements specifically identified by the "and / or" clause, other elements may optionally be present, whether related or unrelated to the specifically identified elements. Thus, as a non-limiting example, a reference to "A and / or B" when used in conjunction with open language such as "comprising" may refer to only A (optionally including elements other than B) in one embodiment; to only B (optionally including elements other than A) in another embodiment; to both A and B (optionally including other elements) in yet another embodiment; and so on.

[0201] As used in the specification and claims, "or" should be understood to have the same meaning as "and / or" defined above. For example, when separating items in a list, "or" or "and / or" should be interpreted as inclusive, i.e., including at least one of several elements or lists of elements, but also including more than one element, and optionally, additional unlisted items. Only terms that clearly indicate the opposite, such as "only one" or "exactly one", or when used in the claims, "consisting of..." will refer to exactly one element in a plurality or series of elements. In general, when preceded by an exclusive term, such as "any", "one of", "only one of", or "exactly one of", the term "or" used herein should only be interpreted as indicating an exclusive alternative (i.e., "one or the other, but not both"). When used in the claims, "essentially consisting of..." should have its ordinary meaning used in the field of patent law.

[0202] If used herein, the terms "approximately" and "about" may be interpreted to mean within 20% of a target value in some embodiments, within 10% of a target value in some embodiments, within 5% of a target value in some embodiments, and within 2% of a target value in some embodiments. The terms "approximately" and "about" may be equal to a target value.

[0203] If used herein, the term "substantially" may be interpreted to mean within 95% of a target value in some embodiments, within 98% of a target value in some embodiments, within 99% of a target value in some embodiments, and within 99.5% of a target value in some embodiments. In some embodiments, the term "substantially" may be equal to 100% of a target value.

[0204] Having described several embodiments of the present invention in detail, it will be readily apparent to those skilled in the art that various modifications and improvements may be present. These modifications and improvements are intended to fall within the spirit and scope of the present invention. Therefore, the foregoing description is intended to be exemplary only and is not intended to be limiting. The present invention is limited only by the appended claims and their equivalents.

Claims

1. A method performed by a computerized system for enabling a user to interact with a physical object in an extended reality environment generated by an extended reality system based at least in part on neuromuscular signals, the method comprising: obtaining a plurality of neuromuscular signals from a user by sensing the plurality of neuromuscular signals using a plurality of neuromuscular sensors disposed on one or more wearable devices worn by the user; determining, using at least one computer processor, information regarding a user's interaction with the physical object in the extended reality environment generated by the extended reality system based at least in part on the plurality of neuromuscular signals, the at least one computer processor coupled to a memory storing code executed by the at least one computer processor; and instructing, using the at least one computer processor, the extended reality system to provide feedback based at least in part on the information regarding the user's interaction with the physical object; Instructing the extended reality system to provide the feedback to the user; wherein: The feedback provided to the user includes an indication of an amount of force applied by the user to the physical object; and The amount of force is determined based at least in part on the plurality of neuromuscular signals and image data associated with the force applied to the physical object, wherein the image data includes images captured using visible light and invisible light.

2. The method according to claim 1, further comprising: Using the at least one computer processor, instructing the extended reality system to display at least one visual indicator on the physical object in the extended reality environment.

3. The method according to claim 2, wherein: Determining the information about a user's interaction with the physical object is based at least in part on a user's interaction with the at least one visual indicator displayed on the physical object.

4. The method according to claim 1, further comprising: capturing, using at least one camera, at least one image of at least a portion of a user, or at least a portion of the physical object, or at least a portion of the user and at least a portion of the physical object, Wherein determining the information about the user's interaction with the physical object is based at least in part on the at least one image. 5 . The method of claim 1 , wherein the indication instructs the extended reality system to provide the feedback as visual feedback to a user in the AR environment. 6 . The method of claim 5 , wherein the indication instructs the extended reality system to provide the visual feedback to the user in the AR environment as a change in at least one property of at least one visual indicator displayed by the AR system. 7 . The method of claim 5 , wherein the visual feedback is provided to the user in the extended reality environment as a display of at least one new visual indicator associated with the physical object in the extended reality environment.

8. The method of claim 1, wherein the instruction instructs the extended reality system to provide the feedback to the user as audio feedback, or electrical stimulation feedback, or audio feedback and electrical stimulation feedback.

9. The method according to claim 1, further comprising: capturing at least one image using at least one camera, Wherein an amount of the force applied to the physical object is determined based at least in part on the at least one image.

10. The method of claim 1, wherein the indication instructs the extended reality system to provide the feedback as a change to the at least one function of the physical object when the at least one function is used in the extended reality environment.

Images