System and method for remotely controlling devices in an environment

Abstract

Provided are computerized systems, methods, apparatus, and computer-readable storage media for generating a 3D map of an environment and / or for utilizing the 3D map to enable a user to control smart devices in the environment and / or interact with people in the environment. A user can wear multiple neuromuscular sensors to generate a 3D map, perform control, and / or interact with people. The sensors can be disposed on a carrier worn by the user and can be configured to sense neuromuscular signals from the user. A camera configured to capture information about the environment can be disposed on the carrier worn by the user. The sensors and camera provide data to a computer processor coupled to a memory.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to and the benefit of U.S. Provisional Patent Application No. 62 / 940,121, filed on November 25, 2019, which is incorporated herein by reference in its entirety. Field of the Invention

[0003] The present technology generally relates to systems and methods for interacting with an environment in which objects can be remotely controlled. The environment can be a real-world environment or an extended reality (XR) environment, such as an augmented reality (AR) environment, a virtual reality (VR) environment, and / or a mixed reality (MR) environment. More specifically, the present technology relates to systems and methods for enabling generation of a three-dimensional (3D) map of an environment based on a user's neuromuscular activity, and systems and methods for utilizing such a 3D map to enable a user to perform control operations in the environment via the user's neuromuscular activity and to perform interactions with objects and / or one or more other people in the environment via the user's neuromuscular activity.

[0004] background

[0005] The real-world environment can be remotely controlled via so-called "Internet of Things" (IoT) technology. Typically, controllable objects (referred to herein as "smart devices") are connected to a network (e.g., the Internet or a dedicated local area network (LAN) of the environment) and are controlled by signals transmitted to the smart device via the network (wirelessly or via hard wiring). Without physically touching the smart device, the user can control the smart device via commands input into a computer, smartphone, tablet, etc. and / or via, for example, (Apple Inc., Cupertino, California, USA), (Amazon.com, Inc., Seattle, Washington, USA) and the like (collectively referred to herein as "Siri / Alexa") to control smart devices. In order to input instructions, multiple steps may need to be performed to access a specific control interface so that instructions can be input for the smart device. As will be understood, control of different smart devices may require access to different control interfaces. It will also be understood that voice instructions may be inconvenient in situations where background noise (e.g., at a loud party) requires shouting out voice commands in order to be received by Siri / Alexa, or in situations where quiet is desired (e.g., when recording a piano performance).

[0006] XR environment systems provide users with an interactive experience of a real-world environment supplemented with virtual information, where computer-generated perceptual information or virtual information is superimposed on various aspects of the real-world environment. Various technologies exist for controlling the operation of XR systems used to generate XR environments and for interacting with XR environments. Current technologies for controlling the operation of XR systems and / or interacting with XR environments have numerous drawbacks, and thus improved technologies are needed.

[0007] Overview

[0008] According to the present invention, a computerized system for remotely controlling a device is provided. The system includes: a plurality of neuromuscular sensors; at least one camera; and at least one computer processor. The plurality of neuromuscular sensors are configured to sense neuromuscular signals from a user and are arranged on at least one wearable device, which is configured to be worn by the user to obtain the neuromuscular signals. The at least one camera is configured to capture information about an environment. The at least one computer processor is programmed to access map information of the environment based on the information about the environment captured by the at least one camera. The map information includes information for controlling at least one controllable object in the environment. The at least one computer processor is also programmed to control at least one controllable object from a first state to a second state in response to neuromuscular activity (optionally, predetermined neuromuscular activity or target neuromuscular activity) identified from the neuromuscular signals sensed by the plurality of neuromuscular sensors.

[0009] Optionally, map information of the environment is stored in a memory. Optionally, the at least one computer processor retrieves the map information from the memory based on information discerned from information about the environment captured by the at least one camera.

[0010] Optionally, the identified information is visible in the environment. Optionally, the identified information includes at least one of: a QR code; a graphic symbol; an alphanumeric text string; a 3D object with a specific shape; and / or a physical relationship between at least two objects.

[0011] Optionally, the identified information includes reference objects of the environment.

[0012] Optionally, the map information includes map data representing a physical relationship between two or more controllable objects in the environment.

[0013] Optionally, the map data is 3D panoramic data of objects in the environment, the objects including at least one controllable object. Optionally, the 3D panoramic data includes a 360° representation of the environment. Optionally, the 3D panoramic data includes data about a single rotational axis. Alternatively or additionally, the 3D panoramic data may include a representation of a partial view of the environment.

[0014] Optionally, the environment is an extended reality (XR) environment including virtual objects and real-world objects. Optionally, the at least one computer processor is programmed to determine position information of the virtual objects and position information of the real-world objects based on the map information, and optionally, determine a reference object in the environment based on information about the environment captured by at least one camera. Optionally, the map information includes position information of at least one controllable object relative to the reference object.

[0015] Optionally, the neuromuscular activity discerned from the neuromuscular signals sensed by the plurality of neuromuscular sensors is caused by activation of a specific motor-unit by the user relative to the at least one controllable object while the user is in the environment.

[0016] Optionally, the neuromuscular activity discerned from the neuromuscular signals sensed by the plurality of neuromuscular sensors is caused by the user performing at least one gesture relative to at least one controllable object while the user is in the environment.

[0017] Optionally, at least one gesture may include one or more of: the user moving at least one finger relative to at least one controllable object; the user moving the wrist relative to at least one controllable object; the user moving the arm relative to at least one controllable object; the user applying force but not moving relative to at least one controllable object; and / or the user activating a motor unit but not moving relative to at least one controllable object and having no force relative to at least one controllable object.

[0018] Optionally, the at least one gesture includes at least one of: the user performing a pinching motion relative to the at least one controllable object using two or more fingers; the user tilting the wrist up or down relative to the at least one controllable object; and / or the user moving at least one finger up or down relative to the at least one controllable object.

[0019] Optionally, the at least one controllable object comprises a plurality of controllable objects. Optionally, the at least one gesture comprises a gesture relative to a controllable object in the plurality of controllable objects. Optionally, the at least one computer processor is programmed to control each of the plurality of controllable objects to change from a first state to a second state in response to a control signal.

[0020] According to the present invention, a wearable electronic device is also provided, which includes: a plurality of neuromuscular sensors configured to sense neuromuscular signals from a user, the plurality of neuromuscular sensors being arranged on at least one wearable device configured to be worn by a user to obtain the neuromuscular signals; at least one camera capable of capturing information about an environment; and at least one computer processor, the at least one computer processor being programmed to access map information of the environment based on the information about the environment captured by the at least one camera, the map information including information for controlling at least one controllable object in the environment, and the at least one computer processor being programmed to control at least one controllable object from a first state to a second state in response to neuromuscular activity discerned from the neuromuscular signals sensed by the plurality of neuromuscular sensors.

[0021] Optionally, the environment is at least one of: a room in a home; a room in a business; a floor of a multi-story building; and / or an outdoor area.

[0022] Optionally, the at least one controllable object comprises at least one of: a display device; an electronic game; a curtain; a light; a sound system; a lock; and / or food or beverage preparation equipment.

[0023] According to the present invention, a computer-implemented method is also provided, which includes: activating multiple neuromuscular sensors configured to sense neuromuscular signals from a user, the multiple neuromuscular sensors being arranged on at least one wearable device configured to be worn by the user to obtain the neuromuscular signals; activating at least one camera capable of capturing information about an environment; accessing map information of the environment based on the information about the environment captured by the at least one camera, the map information including information for controlling at least one controllable object in the environment; and controlling at least one controllable object from a first state to a second state in response to neuromuscular activity identified from the neuromuscular signals sensed by the multiple neuromuscular sensors.

[0024] An electronic device, optionally comprising: a wearable carrier; a plurality of neuromuscular sensors attached to the carrier; a camera system; and at least one computer processor configured to electronically communicate with the plurality of neuromuscular sensors and the camera system.

[0025] According to various aspects of the technology described herein, a computerized system for performing an interaction is provided. The system includes: a plurality of neuromuscular sensors, a camera system, and at least one computer processor. The plurality of neuromuscular sensors can be configured to sense neuromuscular signals from a user and can be arranged on at least one wearable device worn by the user to obtain the neuromuscular signals. The camera system can be configured to capture information about an environment and can include an imaging portion and a depth determination portion. The at least one computer processor can be programmed to: receive captured information from the camera system and receive neuromuscular signals from the plurality of neuromuscular sensors; discern an environment from the captured information; access control information associated with the environment discerned from the captured information, the control information including information for performing at least one function associated with the environment; and, in response to a predetermined neuromuscular activity or a target neuromuscular activity discerned from the neuromuscular signals, cause the at least one function to be performed. Methods and computer-readable storage media storing executable code for implementing these methods are also provided for these aspects.

[0026] A computerized system for obtaining a 3D map, optionally comprising: a plurality of neuromuscular sensors; at least one camera; and at least one computer processor. The plurality of neuromuscular sensors may be configured to sense neuromuscular signals from a user and may be arranged on at least one wearable device configured to be worn by the user to obtain the neuromuscular signals. The at least one camera may be configured to capture information about objects in an environment. The at least one computer processor may be coupled to a memory and may be programmed to generate a 3D map of the environment based on the information captured by the at least one camera and cause the 3D map to be stored in the memory. The 3D map may include information identifying objects in the environment. For example, the identified objects may be smart devices. Methods corresponding to the computerized system for obtaining a 3D map and computer-readable storage media storing executable code for implementing these methods are described.

[0027] It should be appreciated that all combinations of the aforementioned concepts and the additional concepts discussed in more detail below (assuming such concepts are not mutually inconsistent) are contemplated as part of the inventive technology disclosed herein. In addition, all combinations of the subject matter ultimately claimed in this specification are contemplated as part of the inventive technology disclosed herein. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Various non-limiting embodiments of the technology are described with reference to the following drawings. It should be appreciated that the drawings are not necessarily drawn to scale.

[0030] Figure 1is a block diagram of a computer-based system for processing sensor data and camera data, such as sensed signals obtained from a neuromuscular sensor and image data obtained from a camera, according to some embodiments of the technology described herein;

[0031] Figure 2A-2D Schematically illustrates a patch-type wearable system with sensor electronics incorporated thereon according to some embodiments of the technology described herein;

[0032] Figure 3 shows a wearable system having a neuromuscular sensor disposed on an adjustable strap according to some embodiments of the technology described herein;

[0033] Figure 4A shows a wearable system with 16 neuromuscular sensors arranged circumferentially around a band according to some embodiments of the technology described herein; and Figure 4B It passes through Figure 4A A cross-sectional view of one of the 16 neuromuscular sensors shown;

[0034] Figure 5 is a block diagram illustrating components of a computer-based system including a wearable portion and a dongle portion according to some embodiments of the technology described herein;

[0035] Figure 6 schematically illustrates a camera that may be used in one or more systems according to some embodiments of the technology described herein;

[0036] Figure 7 is a diagram schematically illustrating an example implementation of a camera and a wearable system having a neuromuscular sensor disposed on an arm according to some embodiments of the technology described herein;

[0037] Figure 8A is a diagram schematically illustrating another example implementation of a wearable system having a camera and a neuromuscular sensor disposed on an arm, according to some embodiments of the technology described herein;

[0038] Figure 8B and Figure 8C Schematically illustrates some embodiments of the technology described herein. Figure 8A The vertical and axial orientations of the camera;

[0039] Figure 8D shows a wearable system including a camera that can rotate according to some embodiments of the technology described herein;

[0040] Figure 9 schematically illustrates a living room environment in which smart devices are located according to some embodiments of the technology described herein;

[0041] Figure 10 is a block diagram of a distributed computer-based system that integrates an XR system with a neuromuscular activity system according to some embodiments of the technology described herein;

[0042] Figure 11 A flowchart illustrating a process according to some embodiments of the technology described herein, in which neuromuscular signals and camera data are used to capture information of an environment to generate a 3D map of the environment;

[0043] Figure 12 A flowchart illustrating a process for generating a 3D map that can be used to control smart devices in an environment according to some embodiments of the technology described herein;

[0044] Figure 13 A flow chart illustrating a process according to some embodiments of the technology described herein in which neuromuscular signals and camera data are used in conjunction with a 3D map of an environment to control smart devices in the environment; and

[0045] Figure 14A and Figure 14B A flow diagram illustrating a process according to some embodiments of the technology described herein in which neuromuscular signals and camera data are used to control interactions in an environment, including interactions with another person in the environment.

[0046] Figure 15 is an illustration of exemplary augmented reality glasses that may be used in conjunction with embodiments of the present disclosure.

[0047] Figure 16 is an illustration of an exemplary virtual reality headset that can be used with embodiments of the present disclosure.

[0048] Figure 17 is an illustration of an exemplary haptic device that can be used in conjunction with embodiments of the present disclosure.

[0049] Figure 18 is an illustration of an exemplary virtual reality environment according to an embodiment of the present disclosure.

[0050] Figure 19 is an illustration of an exemplary augmented reality environment according to an embodiment of the present disclosure.

[0051] Detailed description

[0052] The technology disclosed herein provides a mapping system and a mapping method that enable a user to create an electronic 3D map of an environment through a combination of neuromuscular sensing technology and imaging technology. A 3D map can be generated in which objects are mapped in an environment. As described below, the 3D map can include image information as well as position and depth information about the objects. The 3D map can also include additional information, such as information that identifies which objects are remotely controllable objects (i.e., smart devices). The 3D map can also include self-identification information, where objects in the environment can be used as reference objects for the environment, and can also be used as searchable objects for identifying a 3D map corresponding to the environment.

[0053] In some embodiments of the present technology, the 3D map may be a map of a real-world environment and may be used to control one or more smart devices in the real-world environment via neuromuscular activity of a user. In some embodiments, the 3D map may include a map of an XR environment and may include information about virtual objects as well as real-world objects in the XR environment.

[0054] The present technology also provides systems and methods for utilizing a 3D map of an environment to enable a user to remotely control or interact with one or more objects in the environment via the user's neuromuscular activity. For example, in the case of a real-world environment containing multiple objects, certain neuromuscular activities of the user (e.g., pointing a finger, closing a hand to form a fist, rotating a wrist, etc.) can be directed to or used to select a smart device to be controlled (e.g., a remotely controllable curtain), and can also be used to control the smart device (e.g., raising or lowering the curtain).

[0055] In another example, a 3D map of an environment can be used with an XR-based system, as described below, such that the environment is an XR environment. The XR environment can be an AR environment, or a VR environment, or an MR environment, or any other type of environment that enables a user to experience aspects of a real-world environment combined with aspects of a virtual environment. In an XR environment, a user can interact with virtual objects (e.g., drawing on a virtual canvas) via certain neuromuscular activities, and can also interact with real-world smart devices (e.g., to adjust remotely controllable curtains). In some embodiments, a user can interact with another person in a real-world environment or in an XR environment via neuromuscular activities performed by the user.

[0056] In some embodiments of the present technology, neuromuscular signals corresponding to the neuromuscular activity of a user may be sensed by one or more wearable sensors worn by the user, as described in more detail below. The neuromuscular signals may be used to determine information about the user's desired remote interaction with one or more objects in the environment. As described above, the environment may be a real-world environment, or an environment generated by an XR-based system, or a combination of both. Such neuromuscular signals may also be referred to herein as "sensed signals." The sensed signals may be used directly as input to a control system for the environment (e.g., by using motor unit action potentials as input signals), and / or the sensed signals may be processed (including by using an inference model as described herein) for the purpose of determining the movement, force, and / or positioning of a part of the user's body (e.g., a finger, hand, wrist, etc.).

[0057] For example, neuromuscular signals obtained by neuromuscular sensors arranged on a wearable device worn by a user can be used to determine the force (e.g., grasping force) applied by the user to a physical object. Multiple muscle activation states of the user can be identified from the sensed signals and / or from information derived from the sensed signals to provide an improved user experience in the environment. Muscle activation states may include, but are not limited to, static postures or gestures performed by the user, dynamic postures or actions performed by the user, activation states of the user's muscles, tensioning or relaxing of the muscles performed by the user, or any combination of the foregoing. The user's interaction with one or more objects in the environment can take a variety of forms, including, but not limited to, selecting one or more objects, controlling one or more objects, activating or deactivating one or more objects, adjusting settings or features related to one or more objects, etc. The user can also interact with another person in the environment.

[0058] It is understood that user interaction may take other forms permitted by the control system for the environment, and need not be the interactions specifically listed herein. For example, control operations performed in the environment may include control based on activation of one or more individual motor units, for example, control based on a sensed or detected activation state of a user's muscles (such as sensed muscle tension).

[0059] It should be understood that the phrases "sensed," "detected," "obtained," "collected," "sensed and recorded," "measured," "recorded," and the like, when used herein in connection with sensed signals from a neuromuscular sensor, include signals detected by the sensor. It should also be understood that the sensed signals can be stored in non-volatile memory prior to being processed, or processed prior to being stored in non-volatile memory. The sensed signals can be cached prior to being processed. For example, after detection, the sensed signals can be stored as "detected" (i.e., raw) in the memory of the neuromuscular sensor, or the sensed signals can be processed at the neuromuscular sensor prior to storing the sensed signals and / or storing the processed signals in the memory of the neuromuscular sensor, or the sensed signals can be transmitted (e.g., via wireless technology, direct wired connection, and / or other known communication technologies) to an external device for processing and / or storage, or any combination of the foregoing. Alternatively, the sensed signals can be processed and utilized without being stored in non-volatile memory.

[0060] Identification of one or more muscle activation states of a user can allow for a hierarchical or multi-level approach to remotely interacting with objects in an environment. For example, in an XR environment, at a first layer / level, one muscle activation state can indicate that the user is interacting with or intending to interact with an object (e.g., a curtain for a window); at a second layer / level, another muscle activation state can indicate a desired control operation (e.g., to open the curtain); at a third layer / level, yet another activation state can indicate that the user wants to activate a set of virtual controls and / or features for the object (e.g., a set of virtual landscape images for different seasons to appear on the window pane); and at a fourth layer / level, yet another muscle activation state can indicate which of the set of virtual controls and / or features the user wants to use when interacting with the object (e.g., a virtual landscape image for summer). It should be understood that any number of muscle activation states and layers can be used without departing from the scope of this disclosure. For example, in some embodiments, one or more muscle activation states can correspond to concurrent gestures based on the activation of one or more motor units, such as a user's hand flexed at the wrist while the index finger points at an object. In some embodiments, one or more muscle activation states may correspond to a sequence of gestures based on activation of one or more motor units, such as a user's hand grasping an object and lifting the object. In some embodiments, a single muscle activation state may indicate a user's desire to interact with an object and activate a set of controls and / or features for interacting with the object.

[0061] For example, a neuromuscular sensor can sense signals of a user's neuromuscular activity. The sensed signals can be input to a computer processor of a control system, which can use, for example, a trained inference model to identify or detect a first muscle activation state of the user, as described below. The first muscle activation state can correspond to, for example, a first gesture performed by the user and can indicate that the user is interacting or intending to interact with a particular object in the environment (e.g., a light). Optionally, in response to detecting the first muscle activation state, feedback can be provided to identify the interaction with the object indicated by the first muscle activation state. The neuromuscular sensor can continue to sense signals of the user's neuromuscular activity and can determine a second muscle activation state from the sensed signals. In response to identifying the second muscle activation state (e.g., corresponding to a second gesture, which can be the same as or different from the first gesture), the control system can activate a set of virtual controls for the object (e.g., controls for turning the light on or off, selecting a light brightness level, selecting a light color, etc.). The neuromuscular sensor can continue to sense signals of the user's neuromuscular activity and can determine a third muscle activation state, and so on.

[0062] In some embodiments of the present technology, a muscle activation state can be identified at least in part from raw (e.g., unprocessed) sensor signals collected by one or more wearable sensors. In some embodiments, a muscle activation state can be identified at least in part from information based on or derived from the raw sensor signals (e.g., processed sensor signals), wherein the raw sensor signals collected by the one or more wearable sensors are processed using one or more techniques (e.g., amplification, filtering, rectification, and / or other forms of signal processing). In some embodiments, a muscle activation state can be identified at least in part from one or more outputs of a trained inference model that receives the sensor signals (either raw or processed versions of the sensor signals) as input.

[0063] In some embodiments of the present technology, according to one or more techniques described herein, a user's muscle activation state determined based on sensed signals can be used to interact with one or more objects in an environment without requiring the user to rely on cumbersome, inefficient, and / or inconvenient input devices. For example, sensor data (e.g., sensed signals or data derived from these signals) can be obtained from a neuromuscular sensor worn by or mounted on the user, and the muscle activation state can be identified based on the sensor data without requiring the user to carry a controller and / or other input device, and without requiring the user to remember complex button or key manipulation sequences. Moreover, the identification of muscle activation states (e.g., posture, gesture, etc.) can be performed relatively quickly based on the sensor data, thereby reducing the response time and delay associated with issuing control signals to the control system, thereby enabling the user to interact in real time or near real time in the environment.

[0064] As described above, the sensing signals obtained by the neuromuscular sensors placed at locations on the user's body can be provided as input to one or more inference models that are trained to generate spatial information of the rigid body segments of a multi-segment articulated rigid body model of the human body (i.e., a model of the human musculoskeletal system). The spatial information may include, for example, positioning information of one or more segments, orientation information of one or more segments, joint angles between segments, and the like. All or part of the human musculoskeletal system can be modeled as a multi-segment articulated rigid body system, where joints form interfaces between different segments, and joint angles define the spatial relationships between the segments connected in the model. Based on the input and due to training, the inference model can implicitly represent the inferred motion of the articulated rigid body under defined movement constraints. The trained inference model can output information that can be used in a variety of applications, such as applications for rendering a representation of the user's body or a portion thereof in an XR gaming environment, and / or applications that utilize certain muscle activation states to control smart devices in a real-world environment.

[0065] For example, motion data obtained by a single motion sensor positioned on a user (e.g., on the user's wrist or arm) can be provided as input data to a trained inference model. The corresponding output data generated by the trained inference model can be used to determine spatial information about one or more segments of a multi-segment articulated rigid body model of the user. For example, the output data can be used to determine the position and / or orientation of an upper arm segment and a lower arm segment of the user connected by an elbow joint. The output data can be used to determine the angle between the two connected segments via the multi-segment articulated rigid body model of the user. Different types of sensors can be used to provide input data to the trained inference model, as described below.

[0066] In some embodiments of the present technology, sensed signals provided to one or more trained inference models can determine that a user is standing with an outstretched forearm pointing forward. The trained inference model can also determine that the fingers of the outstretched forearm have moved from a relaxed, flexed position to a flexed and pointed position, or that the wrist of the outstretched forearm has flexed upward or downward, or that the outstretched forearm has rotated clockwise or counterclockwise, etc. As described below, the muscle activation states identified from the sensed signals can be used in conjunction with a 3D map of the environment to enable the user, for example, to enter the environment and remotely interact with smart devices via neuromuscular signals. Furthermore, as described below, by orienting the user in the environment (e.g., by positioning a reference object in the environment relative to the user), the user can control and / or interact with multiple different smart devices individually (e.g., by pointing a finger at a window curtain of a smart device in the environment and bending the wrist upward to open the window curtain) or collectively (e.g., by pointing a finger at one of multiple smart device lights in the environment and performing a pinching motion with two or more fingers to dim all lights in the environment). It is understood that the output data from the trained inference model can be used for applications other than those specifically identified herein.

[0067] In some embodiments of the present technology, various muscle activation states can be identified directly from sensed signals. In other embodiments, as described above, muscle activation states can be identified based at least in part on the results of processing sensed signals using one or more trained inference models, which may include hand states, postures, poses, etc. (described below). For example, the trained inference model can output motor unit or muscle activation and / or positioning, orientation, and / or force estimates for segments of a computer-generated musculoskeletal model. As used herein, the term "gesture" can refer to the static or dynamic shape of one or more body parts, including the positioning of one or more body parts and the forces associated with the shape. For example, a gesture can include a discrete gesture (e.g., placing or pressing a palm down on a solid surface or grasping a ball), a continuous gesture (e.g., waving fingers back and forth, grasping and throwing a ball), or a combination of discrete and continuous gestures. A gesture can include a subtle gesture that may be imperceptible to another person, such as by co-contracting opposing muscles or using submuscular activation to slightly tense a joint. When training the inference model, gestures can be defined using an application configured to prompt the user to perform gestures, or alternatively, gestures can be arbitrarily defined by the user. Gestures performed by the user can include symbolic gestures (e.g., gestures that are mapped to other gestures, interactions, or commands based on a gesture vocabulary that specifies mappings). In some cases, hand and arm gestures may be symbolic and used for communication based on cultural standards.

[0068] In some embodiments of the present technology, sensed signals may be used to predict information about the positioning and / or movement of a portion of a user's arm and / or the user's hand, which may be represented as a multi-segment articulated rigid body system having joints connecting multiple segments of the rigid body system. For example, in the case of hand movement, sensed signals obtained by neuromuscular sensors placed at locations on the user's body (e.g., the user's arm and / or wrist) may be provided as input to an inference model that is trained to predict positioning (e.g., absolute positioning, relative positioning, orientation) and force estimates associated with multiple rigid segments in a computer-based musculoskeletal representation associated with the hand when the user performs one or more hand movements. The combination of positioning information and force information associated with segments of the musculoskeletal representation associated with the hand may be referred to herein as the "hand state" of the musculoskeletal representation. As the user performs different movements, the trained inference model can interpret the neuromuscular signals into positioning and force estimates (hand state information), which can be output as control signals to control a smart device in the environment or interact with a smart device in the environment or with another person in the environment. Because the user's neuromuscular signals can be continuously sensed, the user's hand state can be updated in real time, and a visual representation of the user's hand (e.g., within the XR environment) can be rendered in real time based on the current estimate of the user's hand state. It is understood that the estimate of the user's hand state can be used to determine the gesture the user is performing and / or predict the gesture the user will perform.

[0069] The limitation of movement at the joint is determined by the joint type of the connecting segment and the biological structure (e.g., muscle, tendon, ligament) that can limit the range of movement at the joint. For example, the shoulder joint that connects the upper arm segment to the subject's torso and the hip joint that connects the thigh segment to the torso are ball-and-socket joints that allow extension and flexion movement as well as rotational movement. In contrast, the elbow joint that connects the upper arm segment and the lower arm segment (or forearm), and the knee joint that connects the subject's thigh segment and the calf segment, allow a more limited range of motion. In this example, a multi-segment articulated rigid body system can be used to model parts of the human musculoskeletal system. However, it should be understood that although some segments of the human musculoskeletal system (e.g., forearm) can be approximated as rigid bodies in an articulated rigid body system, these segments can each include multiple rigid structures (e.g., the forearm can include ulna and radius), which can make it possible to perform more complex movements that the rigid body model may not explicitly consider within the segment. Thus, models of articulated rigid body systems used with some embodiments of the technology described herein may include segments that represent combinations of body parts that are not strictly rigid bodies. It should be understood that physical models other than the multi-segment articulated rigid body systems discussed herein may be used to model portions of the human musculoskeletal system without departing from the scope of this disclosure.

[0070] Continuing with the example above, in kinematics, a rigid body is an object that exhibits various motion properties (e.g., position, orientation, angular velocity, acceleration). Knowing the motion properties of one segment of a rigid body enables the motion properties of other segments of the rigid body to be determined based on constraints on how the segments are connected. For example, a hand can be modeled as a multi-segment articulated body, with joints in each finger and wrist forming joints between multiple segments in the model. In some embodiments, the movement of segments in a rigid body model can be simulated as an articulated rigid body system, where information about the position (e.g., actual position, relative position, or orientation) of a segment relative to other segments in the model is predicted using a trained inference model.

[0071] For some embodiments of the present technology, the part of the human body approximated by the musculoskeletal representation can be a hand or a combination of a hand and one or more arm segments. As described above, in the musculoskeletal representation, the information used to describe the current state of the positioning relationships between segments, the force relationships of the individual segments or combinations of segments, and the muscle and motor unit activation relationships between the segments is referred to as the hand state of the musculoskeletal representation. However, it should be understood that the technology described herein is also applicable to musculoskeletal representations of body parts other than hands, including but not limited to arms, legs, feet, torso, neck, or any combination of the foregoing.

[0072] In addition to spatial (e.g., positioning and / or orientation) information, some embodiments of the present technology are able to predict force information associated with one or more segments of the musculoskeletal representation. For example, a linear force or a rotational (torque) force applied by one or more segments can be estimated. Examples of linear forces include, but are not limited to, the force of a finger or hand pressing on a physical object such as a table, and the force applied when two segments (e.g., two fingers) are pinched together. Examples of rotational forces include, but are not limited to, the rotational force generated when a segment (such as a wrist or finger) is twisted or bent relative to another segment. In some embodiments, the force information determined as part of the current hand state estimate includes one or more of the following: pinch force information, grip force information, and information about the co-contraction force between muscles represented by the musculoskeletal representation. In some embodiments, the force information can be used to set the speed for controlling the smart device. For example, in the aforementioned curtain example, a light touch of two fingers can be used as an instruction to slowly close the curtains, while a more forceful or stronger pinch of two fingers can be used as an instruction to quickly close the curtains.

[0073] Now turning to the accompanying drawings, Figure 1 A system 100 (e.g., a neuromuscular activity system) according to some embodiments of the technology described herein is schematically illustrated. System 100 may include one or more sensors 110 configured to sense signals resulting from activation of motor units within one or more parts of a human body. Sensors 110 may include one or more neuromuscular sensors configured to sense signals generated from neuromuscular activity in skeletal muscles of a human body. As used herein, the term "neuromuscular activity" refers to neural activation, muscle activation, muscle contraction, or any combination of neural activation, muscle activation, and muscle contraction of spinal motor neurons or units innervating a muscle. The one or more neuromuscular sensors may include one or more electromyography (EMG) sensors, one or more myomyography (MMG) sensors, one or more sonomyography (SMG) sensors, a combination of two or more of EMG sensors, MMG sensors, and SMG sensors, and / or one or more sensors of any suitable type capable of detecting neuromuscular signals. In some embodiments, information related to a user's interaction with an environment corresponding to a 3D map may be determined from the neuromuscular signals sensed by the one or more neuromuscular sensors. As a user interacts with the environment over time, spatial information (e.g., position and / or orientation information) and force information related to movement (visible or covert) can be predicted based on the sensed signals. In some embodiments, neuromuscular sensors can sense muscle activity associated with movement caused by an external object (e.g., movement of a user's hand being pushed by an external object).

[0074] The sensors 110 may include one or more inertial measurement units (IMUs) that measure a combination of physical aspects of motion using, for example, accelerometers, gyroscopes, magnetometers, or any combination of one or more accelerometers, gyroscopes, and magnetometers. In some embodiments, the IMUs may be used to sense data about the movement of a user's body parts to which the one or more IMUs are attached, and information derived from the sensed IMU data (e.g., positioning and / or orientation information) may be tracked as the user moves over time. For example, as the user moves over time, one or more IMUs may be used to track the movement of parts of the user's body near the user's torso (e.g., arms, legs) relative to the IMUs.

[0075] In an embodiment comprising at least one IMU and one or more neuromuscular sensors, the IMU and neuromuscular sensors may be arranged to detect movement of different parts of the human body. For example, the IMU may be configured to detect movement of one or more body segments near the user's torso (e.g., movement of the upper arm), while the neuromuscular sensors may be arranged to detect motor unit activity within one or more body segments distal to the user's torso (e.g., movement of the lower arm (forearm) or wrist). (The terms "lower arm" and "forearm" may be used interchangeably herein). However, it should be understood that the sensors (i.e., IMU and neuromuscular sensors) may be arranged in any suitable manner, and that the embodiments of the technology described herein are not limited to any particular arrangement of sensors. For example, in some embodiments, at least one IMU and multiple neuromuscular sensors may be co-located on a body segment of the user to track motor unit activity and / or movement of the body segment using different types of measurements. In one embodiment, the IMU and multiple EMG sensors may be arranged on a wearable device configured to be worn around the user's lower arm or wrist. In such an arrangement, the IMU can be configured to track movement information (e.g., position and / or orientation) associated with one or more arm segments over time to determine, for example, whether a user has raised or lowered his / her arm, while the EMG sensor can be configured to determine finer-grained or more refined movement information and / or submuscular information associated with activation of muscles or submuscular structures in the muscles of the wrist and / or hand.

[0076] During the performance of a motor task, as muscle tension increases, the firing rates of active neurons increase and additional neurons may become active, a process that can be referred to as motor unit recruitment. The pattern in which neurons become active and increase their firing rates is stereotyped, such that the expected motor unit recruitment pattern can define an activity manifold associated with standard or normal movement. In some embodiments of the present technology, because the pattern of motor unit activation is different from the expected or typical motor unit recruitment pattern, the sensed signal can identify the activation of a single motor unit or a group of motor units that are "off-manifold". This non-manifold activation may be referred to herein as "sub-muscular activation" or "activation of sub-muscular structures," where sub-muscular structures refer to a single motor unit or a group of motor units associated with the non-manifold activation. Examples of off-manifold motor unit recruitment patterns include, but are not limited to, selectively activating higher threshold motor units without activating lower threshold motor units that are typically activated earlier in the recruitment sequence, and regulating the firing rates of motor units across a considerable range without regulating the activity of other neurons that are typically co-regulated in the typical motor unit recruitment pattern. In some embodiments, one or more neuromuscular sensors can be positioned relative to the user's body and configured to sense submuscular activation in the absence of observable movement, i.e., in the absence of readily observable corresponding movement of the user's body. In accordance with some embodiments of the present technology, submuscular activation can be used, at least in part, to interact with objects in a real-world environment and an XR environment.

[0077] Some or all of the sensors 110 may each include one or more sensing components configured to sense information about the user. In the case of an IMU, the sensing components of the IMU may include any one or any combination of the following: an accelerometer, a gyroscope, or a magnetometer, which can be used to measure or sense characteristics of the user's body movements. Examples of characteristics of the user's body movements include, but are not limited to, acceleration, angular velocity, and the magnetic field surrounding the body during the user's body movements. In the case of a neuromuscular sensor, the sensing components may include, but are not limited to, electrodes that detect electrical potentials on the body surface (e.g., electrodes used in EMG sensors), vibration sensors that measure vibrations on the skin surface (e.g., vibration sensors used in MMG sensors), acoustic sensing components that measure ultrasonic signals caused by muscle activity (e.g., acoustic sensing components used in SMG sensors), or any combination thereof. Optionally, the sensors 110 may include any one or any combination of the following: a thermal sensor (e.g., a thermistor) that measures the user's skin temperature; an electrocardiogram (ECG) sensor that measures the user's pulse and / or heart rate; a moisture sensor that measures the user's sweat status; and the like. Exemplary sensors that may be used as part of one or more sensors 110 according to some embodiments of the technology disclosed herein are described in more detail in U.S. Patent No. 10,409,371, entitled “METHODS AND APPARATUS FOR INFERRING USER INTENT BASED ON NEUROMUSCULAR SIGNALS,” which is incorporated herein by reference.

[0078] In some embodiments, the one or more sensors 110 may include a plurality of sensors 110, and at least some of the plurality of sensors 110 may be arranged as part of a wearable device that is configured to be worn on or around a portion of a user's body. For example, in one non-limiting example, the IMU and the plurality of neuromuscular sensors may be circumferentially arranged on an adjustable band (e.g., an elastic band), such as a wristband or armband configured to be worn around a user's wrist or arm, as described in more detail below. In some embodiments, a plurality of wearable devices (each having one or more IMUs and / or one or more neuromuscular sensors included thereon) may be used to determine information related to an interaction between a user and an object based on activation from underlying muscle structures and / or based on movement involving multiple parts of the body. Optionally, at least some of the plurality of sensors 110 may be arranged on a wearable patch that is configured to be attached to a portion of a user's body.

[0079] Figures 2A to 2D Various types of wearable patches are shown. Figure 2AA wearable patch 22 is shown in which circuitry for electronic sensors may be printed on a flexible substrate configured to be adhered to an arm, for example, near a vein, to sense blood flow within the user's body. The wearable patch 22 may be an RFID-type patch that can wirelessly transmit sensed information when interrogated by an external device. Figure 2B A wearable patch 24 is shown in which electronic sensors can be incorporated into a substrate configured to be worn on a user's forehead, for example, to measure moisture from sweat. The wearable patch 24 may include circuitry for wireless communication, or may include a connector configured to connect to a cable (e.g., a cable attached to a helmet, head-mounted display, or another external device). The wearable patch 24 may be configured to adhere to the user's forehead or to be held against the user's forehead by, for example, a headband, a skull cap, or the like. Figure 2C A wearable patch 26 is shown in which circuitry for an electronic sensor may be printed on a substrate configured to be adhered to a user's neck, for example, near the user's carotid artery, to sense blood flow to the user's brain. The wearable patch 26 may be an RFID-type patch or may include a connector configured to connect to external electronics. Figure 2D A wearable patch 28 is shown in which electronic sensors may be incorporated into a substrate configured to be worn near a user's heart, for example, to measure the user's heart rate or to measure blood flow to / from the user's heart. As will be appreciated, wireless communication is not limited to RFID technology, and other communication technologies may be employed. Furthermore, as will be appreciated, sensor 110 may be incorporated into other types of wearable patches that may be used with other wearable patches. Figures 2A to 2D The wearable patches shown in are constructed differently.

[0080] In some implementations of the present technology, the sensor 110 may include sixteen neuromuscular sensors arranged circumferentially around a strap (e.g., an elastic strap, etc., an adjustable strap) configured to be worn around a user's lower arm (e.g., around a user's forearm). For example, Figure 3 An embodiment of a wearable system 300 is shown in which neuromuscular sensors 304 (e.g., EMG sensors) are arranged on an adjustable strap 302. It should be understood that any suitable number of neuromuscular sensors can be used, and the number and arrangement of neuromuscular sensors used can depend on the specific application for which the wearable system 300 is used. For example, a wearable armband or wristband can be used to sense information for controlling a robot, controlling a vehicle, scrolling text, controlling an avatar, or any other suitable control task. In some embodiments, the adjustable strap 302 can also include one or more IMUs (not shown).

[0081] Figure 4A 、 Figure 4B 、 Figure 5 Other embodiments of wearable systems of the present technology are shown. In particular, Figure 4A Shown is a wearable system 400 that includes a plurality of sensors 410 arranged circumferentially around an elastic band 420 configured to be worn around the user's lower arm or wrist. The sensors 410 can be neuromuscular sensors (e.g., EMG sensors). As shown, there can be sixteen sensors 410 arranged circumferentially around the elastic band 420 at regular intervals. It should be understood that any suitable number of sensors 410 can be used, and the spacing is not necessarily regular. The number and arrangement of the sensors 410 can depend on the specific application for which the wearable system is used. For example, when the wearable system is to be worn on the wrist, the number and arrangement of the sensors 410 can be different compared to when worn on the thigh. As described above, wearable systems (e.g., armbands, wristbands, thigh bands, etc.) can be used to sense information for controlling a robot, controlling a vehicle, scrolling text, controlling a virtual avatar, and / or performing any other suitable control task.

[0082] In some embodiments of the present technology, the sensor 410 may include only a set of neuromuscular sensors (e.g., EMG sensors). In other embodiments, the sensor 410 may include a set of neuromuscular sensors and at least one auxiliary device. The auxiliary device may be configured to continuously or intermittently collect one or more auxiliary signals. Examples of auxiliary devices include, but are not limited to, IMUs, microphones, imaging devices (e.g., cameras), radiation-based sensors for use with radiation-generating devices (e.g., laser scanning devices), heart rate monitors, and other types of devices that can capture the condition of a user or other characteristics of a user. Figure 4A As shown, sensors 410 may be coupled together using flexible electronics 1430 incorporated into a wearable system. Figure 4B Shown through Figure 4A FIG. 4 is a cross-sectional view of one of the sensors 410 of the wearable system shown in FIG.

[0083] In some embodiments of the present technology, the output of one or more sensing components of sensor 410 may optionally be processed (e.g., amplified, filtered, and / or rectified) using hardware signal processing circuitry. In other embodiments, software may be used to perform at least some signal processing on the output of the sensing components. Thus, signal processing of the sensing signals detected by sensor 410 may be performed by hardware or software, or by any suitable combination of hardware and software, without the various aspects of the technology described herein being limited in this respect. Figure 5 Non-limiting examples of signal processing procedures for processing recorded data from sensor 410 are discussed in more detail.

[0084] Figure 5 is a block diagram illustrating the internal components of a wearable system 500 having sixteen sensors (e.g., EMG sensors) according to some embodiments of the technology described herein. As shown, the wearable system 500 includes a wearable portion 510 and a dongle portion 520. Although not specifically shown, the dongle portion 520 communicates with the wearable portion 510 (e.g., via Bluetooth or another suitable short-range wireless communication technology). The wearable portion 510 may include the sensors 410, examples of which are described above in conjunction with Figure 4A and Figure 4B Described. The sensor 410 provides an output (e.g., a sensed signal) to an analog front end 530, which performs analog processing (e.g., noise reduction, filtering, etc.) on the sensed signal. The processed analog signal generated by the analog front end 530 is then provided to an analog-to-digital converter 532, which converts the processed analog signal into a digital signal that can be processed by one or more computer processors. An example of a computer processor that can be used according to some embodiments is a microcontroller (MCU) 534. The MCU 534 can also receive inputs from other sensors (e.g., IMU 540) and from a power supply and battery module 542. As will be understood, the MCU 534 can receive data from other devices that are not specifically shown. The processed output of the MCU 534 can be provided to an antenna 550 for transmission to the dongle portion 520.

[0085] The dongle portion 920 includes an antenna 952 that communicates with the antenna 550 of the wearable portion 510. Communication between the antennas 550 and 552 may be performed using any suitable wireless technology and protocol, non-limiting examples of which include radio frequency signaling and Bluetooth. As shown, the signal received by the antenna 552 of the dongle portion 520 may be provided to a host computer for further processing, for display, and / or for implementing control of one or more specific objects (e.g., to perform control operations in an environment with respect to smart devices and other controllable objects that are identifiable in a 3D map of the environment). Although reference is made to Figure 4A 、 Figure 4B and Figure 5 The examples provided are discussed in the context of interfacing with EMG sensors, but it should be understood that the wearable systems described herein may also be implemented with other types of sensors, including but not limited to myocardial isotope (MMG) sensors, sonomyographic (SMG) sensors, and electrical impedance tomography (EIT) sensors.

[0086] Back to Figure 1In some embodiments, the sensed signals obtained by sensor 110 can optionally be processed to compute additional derived measurements, which can then be provided as input to an inference model, as described above in more detail below. For example, the sensed signals obtained from an IMU can be processed to derive an orientation signal that specifies the orientation of a segment of a rigid body over time. Sensor 110 can implement signal processing using components that are integrated with the sensing components of sensor 110, or at least a portion of the signal processing can be performed by one or more other components that communicate with, but are not directly integrated with, the sensing components of sensor 110.

[0087] The system 100 also includes one or more computer processors 112 programmed to communicate with the sensors 110. For example, sensory signals obtained by one or more of the sensors 110 may be output from the sensors 110 (in raw form or processed form as described above) and provided to the processor 112, which may be programmed to execute one or more machine learning algorithms to process the sensory signals. The algorithms may process the sensory signals to train (or retrain) one or more inference models 114, and the trained (or retrained) inference models 114 may be stored for subsequent use in generating selection signals and / or control signals to control objects in the environment of the 3D map, as described below. As will be appreciated, in some embodiments, the inference model 114 may include at least one statistical model.

[0088] In some embodiments of the present technology, the inference model 114 may include a neural network, and for example, may be a recurrent neural network. In some embodiments, the recurrent neural network may be a long short-term memory (LSTM) neural network. However, it should be understood that the recurrent neural network is not limited to being an LSTM neural network, and may have any other suitable architecture. For example, in some embodiments, the recurrent neural network may be any one or any combination of the following: a fully recurrent neural network, a gated recurrent neural network, a recursive neural network, a Hopfield neural network, an associative memory neural network, an Elman neural network, a Jordan neural network, an echo state neural network, and a second order recurrent neural network and / or other suitable types of recurrent neural networks. In other embodiments, a neural network that is not a recurrent neural network may be used. For example, a deep neural network, a convolutional neural network, and / or a feedforward neural network may be used.

[0089] In some embodiments of the present technology, the inference model 114 can generate discrete outputs. For example, when the desired output is to know whether a particular activation pattern (including individual neural spike events) is detected in the sensed neuromuscular signal, a discrete output (e.g., a discrete classification) can be used. For example, the inference model 114 can be trained to estimate whether the user is activating a particular motor unit, activating a particular motor unit with a particular timing, activating a particular motor unit with a particular firing pattern, or activating a particular combination of motor units. Over shorter time scales, discrete classifications can be used in some embodiments to estimate whether a particular motor unit has fired an action potential within a given amount of time. In this case, these estimates can then be accumulated to obtain an estimated firing rate for the motor unit.

[0090] In an embodiment where the inference model is implemented as a neural network configured to output a discrete output (e.g., a discrete signal), the neural network can include an output layer that is a normalized exponential layer, such that the outputs of the inference model sum to 1 and can be interpreted as probabilities. For example, the output of the normalized exponential layer can be a set of values ​​corresponding to a set of corresponding control signals, where each value indicates the probability that the user wants to perform a particular control action. As a non-limiting example, the output of the normalized exponential layer can be a set of three probabilities (e.g., 0.92, 0.05, and 0.03) indicating the corresponding probability that the detected activity pattern is one of three known patterns.

[0091] However, it should be understood that when the inference model is a neural network configured to output discrete outputs (e.g., discrete signals), the neural network is not required to produce outputs that sum to one. For example, the output layer of the neural network can be a sigmoid layer, rather than a normalized exponential layer, which does not restrict the outputs to probabilities that sum to one. In such embodiments, the neural network can be trained with a sigmoid cross-entropy cost. Such an implementation may be advantageous in situations where, for example, multiple different control actions may occur within a threshold amount of time, and distinguishing the order in which these control actions occur is not important (e.g., a user may activate two patterns of neural activity within a threshold amount of time). It should be understood that any other suitable non-probabilistic multi-class classifier may be used, as aspects of the technology described herein are not limited in this respect.

[0092] In some embodiments of the present technology, the output of the inference model 114 may be a continuous signal rather than a discrete output (e.g., a discrete signal). For example, the inference model 114 may output an estimate of the firing rate of each motor unit, or the inference model 114 may output a time series of electrical signals corresponding to each motor unit or underlying muscle structure.

[0093] It should be understood that aspects of the technology described herein are not limited to the use of neural networks, and in some embodiments, other types of inference models may also be employed. For example, in some embodiments, the inference model 114 may include a hidden Markov model (HMM), a switching HMM (where switching allows toggling between different dynamic systems), a dynamic Bayesian network, and / or other suitable graphical models with a time component. Any of these inference models can be trained using the sensory signals obtained by the sensor 110.

[0094] As another example, in some embodiments, the inference model 114 may be or include a classifier that takes as input features derived from the sensed signal obtained by the sensor 110. In such embodiments, the classifier can be trained using features extracted from the sensed signal. The classifier can be, for example, a support vector machine, a Gaussian mixture model, a regression-based classifier, a decision tree classifier, a Bayesian classifier, and / or other suitable classifiers, and the present technology is not limited in this regard. The input data provided to the classifier can be derived from the sensed signal in any suitable manner. For example, the sensed signal can be analyzed as time series data using wavelet analysis techniques (such as continuous wavelet transform, discrete time wavelet transform, etc.), Fourier analysis techniques (such as short-time Fourier transform, Fourier transform, etc.), and / or other suitable types of time-frequency analysis techniques. As a non-limiting example, the sensed signal can be transformed using a wavelet transform, and the resulting wavelet coefficients can be provided to the classifier as input data.

[0095] In some embodiments of the present technology, parameter values ​​of the inference model 114 can be estimated based on the training data. For example, when the inference model 114 is or includes a neural network, the parameters (e.g., weights) of the neural network can be estimated based on the training data. In some embodiments, the parameters of the inference model 114 can be estimated using gradient descent, stochastic gradient descent, and / or other suitable iterative optimization techniques. In embodiments where the inference model 114 is or includes a recurrent neural network (e.g., LSTM), the inference model 114 can be trained using stochastic gradient descent and backpropagation through time. Training can employ a cross entropy loss function and / or other suitable loss functions, and aspects of the present technology are not limited in this respect.

[0096] The system 100 may also include one or more controllers 116. For example, the controller 116 may include a display controller configured to display a visual representation (e.g., a representation of a hand) on a display device (e.g., a display monitor). As discussed herein, the one or more computer processors 112 may implement one or more inference models 114 that receive as input the sensed signals obtained by the sensors 110 and provide as output information (e.g., predicted hand state information) that can be used to generate control signals that can be used to control, for example, a smart device or other controllable object in an environment defined by a 3D map.

[0097] The system 100 may also optionally include a user interface 118. Feedback determined based on the sensed signals obtained by the sensor 110 and processed by the processor 112 can be provided to the user via the user interface 118 to facilitate the user's understanding of how the system 100 interprets the user's muscle activity (e.g., intended muscle movement). The user interface 118 can be implemented in any suitable manner, including but not limited to an audio interface, a video interface, a tactile interface, and an electrical stimulation interface, or any combination thereof.

[0098] The system 100 may have an architecture that may take any suitable form. Some embodiments of the present technology may employ a thin architecture in which the processor 112 is or is included as part of a device separate from and in communication with the sensor 110 that may be disposed on one or more wearable devices. The sensor 110 may be configured to wirelessly stream sensed signals (in raw or processed form) and / or information derived from the sensed signals to the processor 112 for processing in substantially real time. The device separate from and in communication with the sensor 110 may be, for example, any one or any combination of the following: a remote server, a desktop computer, a laptop computer, a smartphone, a wearable electronic device (such as a smartwatch), a health monitoring device, smart glasses, an XR-based system, and a control system whose 3D map can be used to identify smart devices or other controllable objects in the environment.

[0099] Some embodiments of the present technology may employ a thick architecture, wherein processor 112 may be integrated with one or more wearable devices on which sensor 110 is disposed. In yet other embodiments, processing of sensed signals obtained by sensor 110 may be divided among multiple processors, at least one of which may be integrated with sensor 110, and at least one of which may be included as part of a device separate from and in communication with sensor 110. In such implementations, sensor 110 may be configured to transmit at least some of the sensed signals to a first computer processor located remote from sensor 110. The first computer processor may be programmed to train at least one of inference models 114 based on the sensed signals transmitted to the first computer processor. The first computer processor may then be programmed to transmit the at least one trained inference model to a second computer processor integrated with one or more wearable devices on which sensor 110 is disposed. The second computer processor may be programmed to use the at least one trained inference model transmitted from the first computer processor to determine information related to interactions between a user wearing the one or more wearable devices and objects in an environment within a 3D map. In this way, the training process and the real-time process using the trained at least one inference model can be independently performed by using different processors.

[0100] In some embodiments of the present technology, the controller 116 can instruct a computer application of the XR system that simulates the XR environment to provide a visual representation by displaying virtual objects. For example, the virtual object can be a character (e.g., an avatar), an imaginary image (e.g., a scene representing a desired season), a tool (e.g., a paintbrush). In one example, the positioning, movement, and / or applied force of a portion of the virtual character within the XR environment can be displayed based on the output of at least one trained inference model. The visual representation can be dynamically updated by using continuously sensed signals obtained by the sensor 110 and processed by the trained inference model 114 to provide a computer-generated representation of the character's movement that is updated in real time.

[0101] Information obtained by or provided to system 100 (e.g., input obtained from a camera, input obtained from sensors 110, etc.) can be used to improve the user experience when a user interacts with the 3D mapped environment, including the accuracy of interaction and / or control operations, feedback, inference models, calibration functions, and other aspects of system 100. To this end, for an XR environment generated by an XR system operating with system 100, the XR system can include at least one processor, at least one camera, and at least one display that provides XR information within the user's field of view. The at least one display can be user interface 118, a display interface provided to the user via AR glasses, or another viewing device that can be viewed by the user. System 100 can include system elements that couple the XR system with a computer-based system that generates a musculoskeletal representation based on sensor data (e.g., sensory signals from at least one neuromuscular sensor). In some embodiments of the present technology, these systems can be combined as subsystems of system 100. In other embodiments, these systems can be coupled via a dedicated computer system or other type of computer system that receives input from the XR system and the computer-based system and generates an XR musculoskeletal representation based on the input. Such a system may include a gaming system, a robotic control system, a personal computer, or another system capable of interpreting XR information and musculoskeletal information. In some embodiments, the XR system and the computer-based system can be configured to communicate directly with each other such that the computer-based system generates an XR musculoskeletal representation of the XR environment. In this regard, information can be communicated using any number of interfaces, protocols, and / or media.

[0102] In some embodiments of the present technology, the system 100 may include one or more cameras 120 that can be used in conjunction with the sensing signals from the sensors 110 to provide an enhanced user experience in an environment containing smart devices. In various embodiments, such smart devices can be identified via a 3D map of the environment. In various other embodiments, as discussed in more detail below, such smart devices can be identified via sensing signals and information obtained by the cameras 120 to enable generation of a 3D map of such an environment.

[0103] As described above, in some embodiments of the present technology, an inference model can be used to predict information used to generate a computer-based musculoskeletal representation and / or update the computer-based musculoskeletal representation in real time. For example, the predicted information can be predicted hand state information. The inference model can be used to predict information based on IMU signals, neuromuscular signals (e.g., EMG, MMG, and / or SMG signals), camera signals, external or auxiliary device signals (e.g., laser scanning signals), or a combination of these signals when detected while the user performs one or more movements and / or while the user engages in other types of neuromuscular activity. For example, camera 120 can be used with an XR system to capture data on the actual positioning of the user's hand. The captured data can be used to generate a computer-based musculoskeletal representation of the user's hand, and this actual positioning information can be used by the inference model to improve the accuracy of the representation and generate a visual representation (e.g., a virtual hand) in an XR environment generated by the XR system. For example, a visual representation of muscle group activation, forces being applied, objects being lifted via the user's movements, and / or other information related to the computer-based musculoskeletal representation can be rendered in a visual display in the XR environment of the XR system.

[0104] In some embodiments of the present technology, an inference model can be used to map muscle activation state information to control signals, where the muscle activation state information is information identified from sensed neuromuscular signals obtained from a neuromuscular sensor. The inference model can receive as input IMU signals, neuromuscular signals (e.g., EMG, MMG, and SMG signals), camera signals, external or auxiliary device signals, or a combination of these signals that are detected and / or captured when the user performs one or more muscle activations, one or more movements, and / or one or more gestures. The inference model can be used to predict control information without the user having to make perceptible movements.

[0105] According to some embodiments of the present technology, the camera 120 can be used to capture information to improve the interpretation of neuromuscular signals and their relationship to movement, positioning, and force generation, thereby capturing information in response to specific neuromuscular signals, capturing information that can be used to identify an environment corresponding to a 3D map, and / or capturing information for generating a 3D map of an environment. It is to be understood that the captured information can be, for example, an image signal corresponding to an image captured by the camera 120. The camera 120 can include a still camera, a video camera, an infrared camera, a stereo camera, a panoramic camera, etc., which is or is capable of capturing one or more 3D images of the user and / or one or more 3D images of an environment of interest to the user or surroundings of the user. Optionally, the camera 120 can be equipped with one or more filters so that the camera 120 can only capture 3D images of light within a specific wavelength range.

[0106] The information captured by camera 120 may include a sequence of still 3D images (an image sequence) and / or a sequence of one or more 3D moving images (a video sequence), which may be captured as one or more signals; thus, references to captured images should be understood to include captured image signals. The terms "camera information," "camera data," and "camera signal" may be used herein to refer to information about a user and / or information about the user's environment that a camera may capture. It should be understood that although various embodiments may refer to "a" camera or "the" camera, these embodiments may utilize two or more cameras in place of one camera. Furthermore, the camera information may relate to any one or any combination of the following: 3D images generated by visible light, 3D images generated by non-visible (e.g., infrared) light, 3D images generated by light of a specific wavelength range, 3D images generated by light of two or more different wavelength ranges, 3D images generated using stereoscopic techniques, 3D images generated by providing a 2D image with depth information, and the like. For example, non-visible light can be used to capture a 3D image of an object of a user's body having a different thermal distribution relative to other nearby objects (e.g., a heat sink), which can provide an indication of blood flow in the user's body, which in turn can be used to infer the user's condition (e.g., a force being applied by a user's finger may have a different blood flow pattern than a finger not applying force).

[0107] In some embodiments, the camera 120 may include a Figure 6 Schematically shown in FIG. Camera 600 may include an imaging portion 602 configured to capture one or more digital images comprising a plurality of pixels of image data. For example, imaging portion 602 may include a red, green, and blue (RGB) camera and / or a near infrared (NIR) camera. Camera 600 may also include a depth portion 604 configured to detect the distance from camera 600 to one or more surfaces in the camera's field of view. For example, depth portion 604 may include an illumination device 604a (e.g., an infrared (IR) diode) configured to emit IR light and a detector 604b configured to receive IR light reflected from a surface in the field of view. Distance or depth may be determined using known time-of-flight techniques. Depth portion 604 and imaging portion 602 may be arranged to capture image data and detect depth data in the same field of view such that each pixel of the image data may be provided with corresponding depth data.

[0108] The camera 600 can be mounted on the user's head (e.g., on a headband, hat, cap, helmet, glasses, etc.) so that the user's head can be used to aim the camera 600 in a desired direction to capture an image. Alternatively, the camera 600 can be mounted on the user's arm (e.g., on a glove, wristband, armband, etc.) so that the user's arm can be used to aim the camera 600 in a desired direction to capture an image. The image can be captured as a still image or as a scan of a video image. The camera 600 can also include other types of lights, bulbs, or lamps, including but not limited to halogen, UV, black light, incandescent, metal halide, fluorescent, neon, and / or light-emitting diodes (LEDs). The camera 600 can communicate with an onboard processor and / or a remote processor so that the captured image can be processed, for example, on an armband worn by the user or via a remote computer or processing unit in communication with the armband.

[0109] The camera 120 (e.g., camera 600) may include circuitry (e.g., a controller) configured to receive control signals from the processor 112 based on one or more neuromuscular activation states determined from the sensed signals. For example, a first activation state may be discerned by the processor 112 as a user desiring to capture an image or initiate video scanning; a second activation state may be discerned by the processor 112 as a user desiring to stop video scanning; a third activation state may be discerned by the processor 112 as a user desiring to identify a specific object (e.g., a controllable object in a smart device or other object); a fourth activation state may be discerned by the processor 112 as a user desiring to control a designated smart device or controllable object to perform a specific function; and a fifth activation state may be discerned by the processor 112 as a user desiring to perform an interaction with another person. The aforementioned activation states may occur in any order or may be independent steps. It should be understood that the camera 120 and the processor may communicate wirelessly (e.g., via Bluetooth technology, near-field communication (NFC) technology, etc.) or via a wired connection.

[0110] Figure 7is a schematic diagram illustrating an example implementation of a system 700 utilizing one or more EMG sensors 740 and a camera 760 in accordance with some embodiments of the technology described herein. For example, system 700 may include system 100. A user's arm 702 and the user's hand 704 are connected and may include an arm / hand portion 710 of the user's body. Arm / hand portion 710 includes multiple joints and segments, which may be depicted as a musculoskeletal representation. More specifically, the user's hand segment 720 is connected by a joint. Any one or any combination of arm positioning, hand positioning, and segment lengths of arm 702 and hand 704 may be determined by system 700 and positioned within the three-dimensional space of a model musculoskeletal representation of arm / hand portion 210. Furthermore, in addition to hand segment 720, the musculoskeletal representation of the user's arm / hand portion 710 may include a forearm segment 730. The system 700 can be used to determine one or more musculoskeletal representations of a user's arm / hand portion 710, which can be used to determine one or more positions of the arm / hand portion 710. To this end, the user can wear a band that includes an EMG sensor 740 that senses the user's neuromuscular signals for use in determining the musculoskeletal representations. While the EMG sensor 740 is sensing the neuromuscular signals, a camera 760 can be used to capture objects within the camera's field of view 750. For example, in Figure 7 In the example, the camera's field of view 750 can be in the same general extension direction of the user's arm / hand portion 710 and can include a portion of the user's arm / hand portion 710. In this example, as described above, the camera 760 can be mounted on the user's head so that the user can change the camera's field of view 750 through head movement. In addition to the sensed signals obtained by the EMG sensor 740, the data captured by the camera 760 can be used to generate a 3D map of the environment, identify smart devices in the environment, control one or more smart devices in the environment, interact with another person in the environment, etc. In addition, the system 700 can render a representation of the user's arm / hand portion 710, such as within an AR environment, based on the sensed signals.

[0111] Figures 8A-8D An embodiment of the present technology is schematically illustrated, wherein a wearable system 800 includes a plurality of neuromuscular sensors 810 (e.g., EMG sensors) and a camera 820 (e.g., camera 600) disposed on an armband 812 configured to be worn on a user's arm 814. Optionally, an IMU, GPS, and / or other auxiliary devices (not shown) may be disposed on the armband 812 along with the camera 820 and neuromuscular sensors 810.

[0112] exist Figure 8B, camera 820 is shown in a vertical orientation to capture images perpendicular to the user's arm 814. When the user is standing on the ground, the user's arm 814 is held directly outward from the user's torso so that when the user's arm 814 is parallel to the ground, the armband 812 can be rotated on the user's arm so that the camera 820 can face upward to capture images of the ceiling of the environment, that is, the camera 820 can have a field of view pointing upward from the user's arm 814. Based on a mapping of the environment (which may include the ceiling and features above, beside, below, and / or in front of the user), embodiments disclosed herein can employ geometric techniques to orient the armband 812 and camera 820 in the environment and thus be able to identify a specific spatial location in the environment around the user at any given time (e.g., even in front of the user when the camera is pointed orthogonally to the plane formed by the user's arms).

[0113] In some embodiments of the present technology, the camera 820 may be arranged to pivot about a hinge 816 or other type of pivoting device. For example, the hinge 816 may enable the camera 820 to be moved from Figure 8B Vertical orientation shown (see also Figure 8A ) adjusted to Figure 8C , which is 90° from the vertical orientation. In the axial orientation, when the user is standing on the ground, the user's arm 814 is held directly outward from the user's torso so that the user's arm 814 is parallel to the ground, and the field of view of the camera 820 can be roughly aligned with the longitudinal direction of the user's arm 814. Therefore, when the camera 820 is in the axial orientation, the user can easily use a finger on the arm 814 to point forward at an object in the field of view of the camera 820. Figure 8D The double-headed arrow in shows that the wearable system 800 including the camera 820 can be rotated around the user's arm 814.

[0114] In an example implementation, a user may use the camera 820 to capture images and / or video to generate an environment (e.g., in Figure 9900) by standing at a central location within the environment and sweeping arm 814 in an arc while the torso is in a fixed position; holding arm 814 in a fixed position relative to the torso and rotating the torso through an angle of 0° to 360° at the central location; walking around the perimeter of the environment while aiming the camera 820's field of view away from the perimeter and inward; and randomly changing the camera's field of view while walking through the environment. Images / videos captured by camera 820 can include still images with a standard aspect ratio; still images with panoramic, wide-angle, or other non-standard aspect ratios; and / or one or more video sweeps. As described above, via neuromuscular sensor 810, the user can control camera 820 using various gestures to inform information to be used to generate a 3D map of living room 900, which can include the entire living room 900 (including walls, ceiling, and floor). For example, a first gesture corresponding to a first activation state may be performed to cause camera 820 to capture a still image or initiate video scanning; a second gesture corresponding to a second activation state may be performed to cause camera 820 to stop video scanning; a third gesture corresponding to a third activation state may be performed to cause a specific object (e.g., lamp 902, television 904, curtains 908, etc.) in the field of view of camera 820 to be identified as a smart device in captured camera data corresponding to living room 900; and a fourth gesture corresponding to a fourth activation state may be performed to cause the object in the field of view of camera 820 to be designated as reference object 906 for living room 900. It will be appreciated that a 3D map generated for living room 900 may be identified based on reference object 906. Optionally, multiple reference objects may be designated for the environment of the 3D map (e.g., a primary reference object and a secondary reference object to ensure proper correlation between the environment and the 3D map).

[0115] In some embodiments of the present technology, images / videos captured by camera 120 while a user is in an environment can be processed by computer processor 112 to identify one or more reference objects in the environment. If a reference object of the environment is identified, the processor can access storage device 122 to retrieve a 3D map of the identified environment. In addition, processor 112 can also activate a control interface for the identified environment to enable the user to interact with smart devices in the identified environment via neuromuscular activation states (e.g., gestures, movements, etc.). Therefore, when a reference object of the environment is identified and a corresponding 3D map is retrieved, the smart devices in the environment can be controlled via the user's neuromuscular activity, instead of using a traditional interface to control the smart devices in the identified environment (e.g., a traditional IoT-type smartphone interface). In an embodiment, storage device 122 can store multiple different maps for multiple different environments. In another embodiment, storage device 122 can store multiple maps for a single environment, where each map identifies a different set of smart devices that the user can use. For example, user A may be allowed to control lights and a sound system through neuromuscular activity via map A corresponding to control interface A, while user B may be allowed to control lights, a sound system, and a television through neuromuscular activity via map B corresponding to control interface B.

[0116] In some embodiments of the present technology, 3D maps may be utilized for XR environments. Figure 10 FIG1 shows a schematic diagram of an XR-based system 1000, which can be a distributed computer-based system that integrates an XR system 1001 with a neuromuscular activity system 1002. The neuromuscular activity system 1002 can be the same as that described above. Figure 1 The system 100 described is the same as or similar thereto.

[0117] The XR system 201 can take the form of a pair of goggles or glasses, or eye protection, or other type of display device that displays display elements to the user that can be superimposed on the user's "reality." In some cases, the reality can be the user's observation of the environment (e.g., as viewed through the user's eyes), or a captured version of the user's observation of the environment. For example, the XR system 1001 can include one or more cameras 1004, which can be mounted within a device worn by the user and capture one or more observations experienced by the user in the user's environment. The XR system 1001 can include one or more processors 1005, which operate within the device worn by the user and / or within a peripheral device or computer system, and such processors 1005 can be capable of sending and receiving video information and other types of data (e.g., sensor data).

[0118] The XR system 1001 may also include one or more sensors 1007, such as any one or any combination of the following: a microphone, a GPS element, an accelerometer, an infrared detector, a tactile feedback element, and the like. In some embodiments of the present technology, the XR system 1001 may be an audio-based or auditory XR system, and the sensors 1007 may also include one or more headphones or speakers. In addition, the XR system 1001 may also include one or more displays 1008, which, in addition to providing the user with a view of the user environment rendered by the XR system 1001, allow the XR system 1001 to overlay and / or display information to the user. The XR system 1001 may also include one or more communication interfaces 1006, which enable information to be transmitted to one or more computer systems (e.g., a gaming system or other system capable of rendering or receiving XR data). XR systems can take many forms and are available from many different manufacturers. For example, various embodiments may be implemented in association with one or more types of XR systems or platforms, such as the HoloLens holographic reality glasses available from Microsoft Corporation (Redmond, WA, USA), the Lightwear AR headset available from Magic Leap (Plantation, FL, USA), the Google Glass AR glasses available from Alphabet (Mountain View, CA, USA), the R-7 Smartglasses System available from Osterhout Design Group (also known as ODG; San Francisco, CA, USA), the Oculus Quest, Oculus Rift S, and Spark AR Studio available from Facebook (Menlo Park, CA, USA), or any other type of XR device.

[0119] XR system 1001 can be operatively coupled to neuromuscular activity system 1002 via one or more communication schemes or methods, including but not limited to Bluetooth protocol, Wi-Fi, Ethernet-like protocols, or any number of wireless and / or wired connection types. It should be understood that, for example, systems 1001 and 1002 can be directly connected or coupled through one or more intermediate computer systems or network elements. Figure 10 The double arrows in represent the communicative coupling between systems 1001 and 1002.

[0120] As described above, the neuromuscular activity system 1002 can be structurally and functionally similar to the system 100. Specifically, the neuromuscular activity system 1002 can include one or more neuromuscular sensors 1009, one or more inference models 1010, and can create, maintain, and store one or more musculoskeletal representations 1011. In an exemplary embodiment, similar to the embodiments discussed above, the neuromuscular activity system 1002 can include or can be implemented as a wearable device, such as a band that can be worn by a user to collect (i.e., obtain) and analyze neuromuscular signals from the user. In addition, the neuromuscular activity system 1002 can include one or more communication interfaces 1012, which allow the neuromuscular activity system 1002 to communicate with the XR system 1001, such as via Bluetooth, Wi-Fi, or other communication means. Notably, the XR system 1001 and the neuromuscular activity system 1002 can transmit information that can be used to enhance the user experience and / or allow the XR system 1001 to operate more accurately and efficiently.

[0121] In some embodiments, the XR system 1001 or the neuromuscular activity system 1002 may include one or more auxiliary sensors configured to record auxiliary signals, which may also be provided as input to one or more trained inference models, as described above. Examples of auxiliary sensors include an IMU, a GPS, an imaging device, a radiation detection device (e.g., a laser scanning device), a heart rate monitor, or any other type of biosensor capable of sensing biophysical information from a user during the execution of one or more muscle activations. In addition, it should be understood that some embodiments of the present technology may be implemented using camera-based systems that perform skeletal tracking, such as the Kinect system available from Microsoft Corporation (Redmond, Washington, USA) and the Leap Motion system available from Leap Motion, Inc. (San Francisco, California, USA). It should also be understood that the various embodiments described herein may be implemented using any combination of hardware and / or software.

[0122] Although Figure 10A distributed computer-based system 1000 is shown that integrates an XR system 1001 with a neuromuscular activity system 1002, but it should be understood that the integration of these systems 1001 and 1002 can be non-distributed in nature. In some embodiments, the neuromuscular activity system 1002 can be integrated into the XR system 1001 so that various components of the neuromuscular activity system 1002 can be considered part of the XR system 1001. For example, the input from the neuromuscular signal sensed by the neuromuscular sensor 1009 can be considered another of the inputs to the XR system 1001 (e.g., from the camera 1004, from the sensor 1007). In addition, the processing of the input (e.g., the sensed signal) obtained from the neuromuscular sensor 1009 can be integrated into the XR system 1001 (e.g., performed by the processor 1005).

[0123] As described above, the present technology relates in certain aspects to a computerized mapping system. The mapping system can generate an electronic three-dimensional (3D) map of an environment (e.g., a room in a house, an office in a building, an indoor warehouse environment, etc.), and the 3D map can identify objects in the environment that can be remotely controlled. The mapping system can include multiple neuromuscular sensors, one or more cameras, one or more computer processors, and one or more memory devices. The neuromuscular sensors can be attached to a wearable device that can be worn by a user to sense neuromuscular signals from the user. As discussed herein, the neuromuscular signals can be processed to determine the user's neuromuscular activity. Neuromuscular activity can be caused by easily visible movements of the user or changes in the user's muscles that may not be easily visible. The computer processor can generate a 3D map based on the neuromuscular signals sensed by the multiple neuromuscular sensors and image information captured by the camera, and can store the 3D map in the memory device. The camera can be controlled to capture one or more images and / or one or more video scans based on the neuromuscular activity identified by the computer processor from the neuromuscular signals. Two or more images and / or two or more video sequences can be stitched together using known image processing techniques.

[0124] For example, a mapping system can generate a 3D map of a real-world room by capturing a video scan of the room and / or capturing one or more still images of the room (collectively, “captured video / images”). The captured video / images can be stored in a memory device. Among the various real-world objects in the captured video / images, the user can identify one or more controllable objects (i.e., smart devices) that can be remotely controlled as so-called “Internet of Things” (IoT) objects. For example, a room may include IoT-controllable lights, an IoT-controllable sound system, and an IoT-controllable video game monitor. The user can identify these IoT-controllable objects via neuromuscular activity detected during or after the captured video / images are captured. In one example, in the case of a video scan, the user can point his / her index finger at each IoT-controllable object while the video scan is being captured. In another example, the user can update a pre-existing 3D map of the room by identifying the IoT-controllable objects via neuromuscular activity.

[0125] More specifically, multiple neuromuscular sensors and cameras can be attached to a wearable device worn on the user's arm. A computer processor can be programmed to control the camera to capture a still image or a video scan when, for example, the user's index finger points. In this example, the user can point with the index finger to start video capture or recording of the room, and can move the arm (and therefore the camera) to scan various parts of the room for video capture. Optionally, the user can identify specific objects of interest in the room (e.g., IoT controllable objects) by performing another neuromuscular activity during video capture. For example, the user can move the index finger up and down to identify objects of interest in the room. Each object of interest can be electronically marked or tagged so that the object can be identified in a 3D map of the room.

[0126] The neuromuscular sensor can output one or more signals to a computer processor and / or a memory device. These signals can be processed by the computer processor to determine each instance of the user moving his / her index finger up and down during the video scan and associate each such instance with an IoT-controllable object. As will be appreciated, each IoT-controllable object can have a corresponding IoT control system (e.g., a corresponding IoT control interface) accessible to the computer processor, such that the computer processor can transmit instruction signals to the IoT-controllable object via the IoT control system.

[0127] To obtain depth information, the camera may include an infrared (IR) distance sensor that includes circuits for IR emission and IR reception. The IR distance sensor may be configured to emit IR light outwardly and receive reflected IR light caused by the emitted IR light impinging on a surface (i.e., a reflective surface) and reflecting back to the IR distance sensor. Using known techniques, a processor and / or a computer processor in the IR distance sensor may determine the distance between each reflective surface and the IR distance sensor based on the time elapsed between the emission of the IR light and the reception of the reflected IR light. It will be understood that the IR distance sensor and the camera may be aligned so that the imaging area of ​​the camera may be associated with the distance information from the IR distance sensor. Using known techniques, each pixel in the pixel array of the image captured by the camera may be associated with the depth or distance information of that pixel.

[0128] Alternatively, the camera may comprise a stereo camera capable of recording 3D still images or 3D video, or may comprise a plurality of cameras mounted relative to each other to obtain image information or video information that may be combined to produce stereoscopic information.

[0129] In certain aspects, the present technology also relates to using the identified neuromuscular activity of a user to control smart devices in an IoT-enabled environment. In this regard, an IoT interaction system is provided that may include a plurality of neuromuscular sensors, one or more cameras, one or more computer processors, and one or more memory devices. The IoT interaction system may be a real-world system or an XR-based system. The neuromuscular sensors may be attached to a wearable device that may be worn by a user to sense neuromuscular signals from the user. The computer processor may process image data captured by the camera to determine whether an object in the image data corresponds to a reference object of a 3D map stored in a memory device. If a match is found, a 3D map corresponding to the reference object may be accessed, and a control interface for a smart device in the environment corresponding to the 3D map may be activated.

[0130] The computer processor can process the sensing signals from the neuromuscular sensors to determine the neuromuscular activity of the user when the user is in an environment corresponding to the 3D map. The identified neuromuscular activity can be used to control smart devices in the environment. If the IoT interaction system is an XR-based system, it can be enabled to have a virtual experience about real-world objects. For example, a user can interact with the real-world windows of a room via neuromuscular activity, for example, to open the curtains that cover the windows. The interaction can be via detection of the user pointing at the curtains and / or via relative movement of the user's fingers to indicate that the curtains are to be opened. The identified predetermined neuromuscular activity can indicate that the user expects to see an animated view through the window (for example, a clockwise rotation of the user's wrist can cause an animated bird to appear in the window, a counterclockwise rotation of the user's wrist can cause an autumn leaf scene to appear in the window, and so on).

[0131] In certain aspects, the present technology also relates to using the neuromuscular activity identified by a user to interact with people in an environment (e.g., an XR environment). In this regard, an XR-based system is provided that may include multiple neuromuscular sensors, one or more cameras, one or more computer processors, and one or more memory devices. The neuromuscular sensors may be attached to a wearable device that can be worn by a user to sense neuromuscular signals from the user. The computer processor may process image data captured by the camera to determine whether an object in the environment is a person. This determination may be based on shape, movement, facial features, etc. Optionally, the computer processor may be equipped with a detector configured to detect signals emitted from a device worn by a person in the environment. If the computer processor determines that a person is present, the computer processor may determine the person's identity. For example, facial recognition processing may be performed on facial features in the captured image data. In another example, signals emitted from a device worn by the user may provide identification of the person. Once the person is identified, the user may use neuromuscular activity to interact with the person in the environment.

[0132] For example, the identified neuromuscular activity could be used to play games with people in the environment, send messages to a person's smartphone, or deliver tactile signals to a person.

[0133] Implementation A - Generation of 3D Maps

[0134] According to some embodiments of implementations of the present technology, a computerized system for obtaining a 3D map of an environment is provided, which may be system 100. The system may include a plurality of neuromuscular sensors, at least one camera, and at least one computer processor. The plurality of neuromuscular sensors may be configured to sense neuromuscular signals from a user. For example, the plurality of neuromuscular sensors may be arranged on at least one wearable device configured to be worn by a user to obtain neuromuscular signals. The at least one camera may be configured to capture information about objects in the environment based on or in response to signals from the plurality of neuromuscular sensors. The at least one computer processor may be coupled to a memory and may be programmed to generate a 3D map of the environment based on or in response to signals from the plurality of neuromuscular sensors or information obtained from the signals from the plurality of neuromuscular sensors, and cause the 3D map to be stored in the memory. The 3D map may include information identifying objects in the environment.

[0135] In various embodiments of this implementation, the system may include GPS circuitry that can provide GPS data associated with the 3D map. For example, the GPS circuitry can be configured to provide GPS coordinates of objects in the environment based on or in response to signals from a plurality of neuromuscular sensors.

[0136] In various embodiments of the present invention, the neuromuscular signals from the plurality of neuromuscular sensors may enable the capture of multiple images of the environment and / or may enable the capture of at least one video scan of the environment. At least one computer processor may generate a 3D map by associating the multiple images or associating multiple portions of the at least one video scan. For example, at least one computer processor may be programmed into a "closed loop mode" and use simultaneous localization and mapping (SLAM) techniques (e.g., visual SLAM or VSLAM, ORB-SLAM, DynaSLAM, etc.) and / or real-time appearance-based mapping (RTAB-Map) techniques to "close the loop" and associate the multiple images or multiple portions of the at least one video scan. Feature tracking algorithms (e.g., scale-invariant feature transform (SIFT), accelerated robust features (SURF), good feature tracking (GFTT), binary robust independent fundamental features (BRIFE), etc.) may be used in conjunction with the SLAM techniques. It is to be understood that other image association techniques known in the art may be used in place of or in conjunction with the techniques identified herein.

[0137] It will be appreciated that some embodiments of the technology described herein may generate or utilize a 3D map of an environment as seen from a predetermined rotation axis, where a camera may rotate through angular sectors (e.g., 90°, 180°, 270°, 360°, etc.) to capture video and / or images, and where depth may be one dimension; these embodiments may involve 3D maps that may be partial maps. In other embodiments of the technology, a 3D map may involve rotation through multiple different rotation axes (e.g., three mutually orthogonal axes); the maps of these embodiments may be generated from video and / or images taken at any angle along multiple different rotation axes from multiple different center points, where a geometric relationship is established between the multiple different center points.

[0138] The at least one computer processor for generating the 3D map may include one or more local processors at the location of the mapped environment and / or one or more remote processors at a location remote from the mapped environment. In some embodiments, the local processor may be located on a wearable device on which the camera and neuromuscular sensor are located (e.g., on a wearable device). Figure 8A On the wearable system 800), or may be located in one or more servers of a local area network (LAN) to which the wearable device belongs. It is to be understood that communication within the LAN can be via any one of the following or any combination thereof: Wi-Fi, 3G, 4G, 5G, Bluetooth, other streaming media transmission technologies, and also traditional hard-wired connections. In some embodiments, the remote processor can be located at a facility that is far away from the wearable device (e.g., in a different city, a different state, a different country, etc.) and can communicate with the wearable device via a global communication network (e.g., the Internet). It is to be understood that while some of the calculations for generating the 3D map may be relatively simple and therefore may require a relatively small amount of computing power and therefore may be performed on a simple processor carried on the wearable device, other calculations may require significantly greater amounts of computing power and therefore a simple processor may not be sufficient. Therefore, it may be advantageous to perform some or all of the calculations for generating the 3D map on one or more dedicated (e.g., high computing power) graphics computers equipped with one or more graphics processing units (GPUs) that can execute. For example, data from neuromuscular sensors and cameras can be streamed or uploaded to a remote cloud facility where it can be accessed by a dedicated computer to generate a 3D map from the data.

[0139] In multiple embodiments of this implementation, at least one computer processor can be programmed to identify a first neuromuscular activity from the neuromuscular signal. Each occurrence of the first neuromuscular activity can cause an image of the environment to be captured or a video scan of the environment to be captured. At least one computer processor can be further programmed to determine a second neuromuscular activity from the neuromuscular signal and associate the second neuromuscular activity with a controllable object (e.g., a smart device) in the environment. A 3D map can be generated such that the 3D map includes information about which object in the environment is a controllable object, so that the controllable object can be identified from the 3D map. That is, where the environment includes multiple controllable objects, each of the controllable objects can be identified on the 3D map.

[0140] In various embodiments of this implementation, the first neuromuscular activity and / or the second neuromuscular activity can include any one or any combination of the following: pointing of a user's finger, unpointing of a user's finger, a user making a fist, a user unmaking a fist, a user clockwise wrist motion, a user counterclockwise wrist motion, a user palm-up gesture, and a user palm-down gesture, or any other suitable arm, finger, hand, or wrist movement or gesture. It will be appreciated that the at least one computer processor can be programmed to discern one or more other types of neuromuscular activity for the first neuromuscular activity and / or the second neuromuscular activity.

[0141] In multiple embodiments of this implementation, the information captured about objects in the environment may include any one of the following or any combination thereof: a visual image of each object in the environment, one or more depth values ​​for each object in the environment, and one or more angle values ​​for each object in the environment. For example, for each object in the environment, the angle value of the object may correspond to the angle between the object and a predetermined origin for the 3D map. The vertex of the angle may correspond to the camera position of at least one camera during the capture of information about the object in the environment. The depth value of the object in the environment may be the line-of-sight distance between at least one camera and the object. Depending on the size of the various objects in the environment, one or more depth values ​​and / or angle values ​​may be used in multiple embodiments disclosed herein.

[0142] In various embodiments of this implementation, at least one camera may be disposed on at least one wearable device. For example, the at least one camera may include a camera disposed on a head-mounted device. The head-mounted device may be one of the following: a headband, a hat, a helmet, and glasses. In another example, the at least one wearable device may include a band configured to wrap around a wrist or forearm of a user. The plurality of neuromuscular sensors may be circumferentially disposed on the band, and the at least one camera may include a camera mounted on the band and disposed radially outside of one or more of the plurality of neuromuscular sensors.

[0143] In various embodiments of this implementation, at least one camera may include a stereo camera capable of capturing 3D images / video.

[0144] In various embodiments of this implementation, at least one camera may include an imaging portion and a depth determination portion. The imaging portion may include an RGB camera. The imaging portion may include at least two optical paths. The depth determination portion may include an infrared light emitter and a receiver. The emitter may be configured to emit infrared light toward one or more surfaces, and the receiver may be configured to receive infrared light reflected from the surfaces.

[0145] In various embodiments of this implementation, information about objects in an environment captured by at least one camera may include multiple images, where each image is formed by a pixel array, and each pixel of the pixel array may include depth data and visual data.

[0146] In multiple embodiments of this implementation, at least one computer processor can be programmed to: identify and mark a specific object among the objects in the 3D map as a reference object of the environment, and identify and mark other objects in the 3D map relative to the reference object, so that identification of the physical location of the reference object in the 3D map can identify the physical locations of the other objects in the 3D map.

[0147] In various embodiments of this implementation, the reference object of the environment may be determined based on any one or any combination of the following: a shape of the reference object, a color or combination of colors of the reference object, a symbol on the reference object, and a surface relief structure on the reference object. For example, when a match is found between a reference object identified with respect to the 3D map and an object in an image captured by the camera, the 3D map may be retrieved from the memory device.

[0148] Figure 11A flowchart of a processing flow 1100 for an embodiment of this implementation is shown. At S1102, a neuromuscular sensor on a user senses a neuromuscular signal of the user and processes the sensed signal. For example, the neuromuscular sensor can be attached to a strap worn around the user's arm. At S1104, if it is determined that the neuromuscular signal includes one or more signals corresponding to a first neuromuscular activity, the processing flow 1100 proceeds to S1106; if not, the processing flow 1100 returns to S1102. For example, the first neuromuscular activity can be the user making a fist to begin imaging the user's current environment. At S1106, a 3D image / video is captured while continuing to sense and process the user's neuromuscular signal. For example, the image / video can be captured by a 3D camera attached to a strap worn around the user's arm.

[0149] At S1108, if it is determined that the neuromuscular signal includes one or more signals corresponding to the second neuromuscular activity, the processing flow 1100 proceeds to S1110; if not, the processing flow 1100 returns to S1106. For example, the second neuromuscular activity can be the user pinching the thumb and index finger together. At S1110, the object that appears in the image / video when the second neuromuscular activity occurs is marked. For example, the user can perform the second neuromuscular activity to indicate a reference object of the environment. The reference object can be an object used to identify the environment relative to other environments. The environment can have one reference object or multiple reference objects. It is to be understood that the field of view of the 3D camera can be aligned with the direction of the user's arm, or can be orthogonal to the direction, or at some other angle to the direction, so that the user's hand or a portion thereof can be captured in the image or can not be captured in the image. Alternatively, if captured in an image, the user's hand can facilitate identification of the object as a reference object; however, as described herein, it is not necessary to capture the user's hand to know where the fingers of the hand are pointing, because neuromuscular signals obtained from the user via one or more neuromuscular sensors (e.g., EMG sensors) on a wearable system worn by the user and image information obtained from a camera on the wearable system can be provided to a trained inference model to determine when and where the fingers of the user's hand are pointing. It should be understood that identification of a specific object can be achieved by extrapolating the pointing direction of the user's finger to the object determined based on the image information. When there are objects in the environment that are far apart from each other, a single extrapolation is sufficient to identify the specific object. On the other hand, when there are multiple objects in the environment that are close to each other, multiple extrapolations may be used to identify the specific object. For example, in the case of multiple extrapolations, each extrapolation can be from a user's finger pointing at the object from a different perspective, and the intersection of the extrapolations can be used to identify the specific object.

[0150] The process then proceeds to S1112, where it is determined whether the neuromuscular signal includes one or more signals corresponding to a third neuromuscular activity. If so, the process 1100 proceeds to S1114; if not, the process 1100 returns to S1106. For example, the third neuromuscular activity may be a user pointing a finger. At S1114, an object that appears in the image when the third neuromuscular activity occurs is marked. For example, the third neuromuscular activity may be performed by the user to indicate a smart device in the environment. The process then proceeds to S1116, where it is determined whether the neuromuscular signal includes one or more signals corresponding to a fourth neuromuscular activity. The one or more signals corresponding to the fourth neuromuscular activity may indicate that the user desires to stop capturing information about the environment. If not, the process 1100 returns to S1106 to enable another reference object and / or another smart device to be marked. If yes, process flow 1100 proceeds to S1118, where sensing and processing of neuromuscular signals ceases, image capture ceases, and sensor data corresponding to the sensed and processed neuromuscular signals and camera data corresponding to the images are stored. The stored sensor data and / or stored camera data may include raw data or processed data, or both. The data may be stored so that the data can be searched based on the labeled reference object.

[0151] Figure 11 The processing flow 1100 may be used to obtain information about an environment to generate a 3D map of the environment.

[0152] Figure 12 A flowchart of a process flow 1200 for another embodiment of this implementation is shown. At S1202, multiple captured images of an environment or portions of one or more video scans of an environment are stitched together to form a 3D map of the environment. For example, from Figure 11 The processing flow 1100 of FIG can be used in this regard. At S1204, the 3D map of the environment is updated to (e.g., based on Figure 11 At S1206, for each smart device identified on the 3D map, a link is formed between the smart device and the control interface for the smart device. Thus, when the 3D map of the environment is accessed for use, each smart device identified in the 3D map is activated to be remotely controllable. At S1208, the 3D map of the environment is updated to (e.g., based on the Figure 11The second neuromuscular activity at S1110 in S1210 identifies each object marked as a reference object. At S1210, the updated 3D map is stored in the memory device so that the 3D map can be searched based on the reference objects of the environment identified on the 3D map. It is to be understood that the link to the smart device can be any means for remotely controlling the smart device using IoT technology. In some embodiments, the link can be a user interface known in the art, which enables communication of instructions from the user to the smart device via a server that processes the instructions and transmits control signals to the smart device. For example, communication can be via Wi-Fi, Bluetooth, LAN, WAN and / or any suitable wired or wireless technology for sending instructions to the server. The server can be programmed to receive instructions from the user interface and transmit appropriate control signals to the smart device. The transmission can be via Wi-Fi, Bluetooth, LAN, WAN and / or any suitable wired or wireless technology for sending control signals to the smart device. Thus, when a 3D map of an environment is accessed for use, a user interface for a smart device in the environment may be accessed and activated for use, such that instructions determined from the user's identified neuromuscular activity may be transmitted via the user interface to one or more servers corresponding to the smart device to process the instructions and control the smart device in the environment in a manner similar to how the smart device is traditionally controlled by instructions received via, for example, a tablet computer, smartphone, and / or one or more other input devices.

[0153] Implementation B - Use of 3D Maps

[0154] According to some embodiments of implementations of the present technology, a computerized system for remotely controlling a device is provided, which may be system 100, or system 1000, or variations of these systems 100, 1000. The system may include a plurality of neuromuscular sensors, at least one camera, and at least one computer processor. The plurality of neuromuscular sensors may be arranged on at least one wearable device configured to be worn by a user to sense neuromuscular signals from the user. The at least one camera may be configured to capture information about an environment of interest to the user. The at least one computer processor may be programmed to access map information of the environment based on the information about the environment captured by the at least one camera. The map information may include information for controlling at least one controllable object in the environment. The at least one computer processor may also be programmed to control the at least one controllable object from a first state to a second state in response to neuromuscular activity discerned from the neuromuscular signals sensed by the plurality of neuromuscular sensors.

[0155] In various embodiments of this implementation, map information of an environment may be stored in a memory, and at least one computer processor may be programmed to retrieve the map information from the memory based on information discerned from information about the environment captured by at least one camera. For example, the discerned information may be visible in the environment and may include any one or any combination of the following: a QR code, a graphical symbol, an alphanumeric text string, a 3D object having a specific shape, and a physical relationship between at least two objects. The discerned information may include reference objects of the environment. For example, if a camera detects a small table and / or a small office lamp, the system may use this information to help identify a given environment as a home office environment; if the camera detects a sofa and / or an ottoman, the system may use this information to help identify the environment as a living room or a family room; if the camera detects architectural lighting and / or ceiling fixtures, the system may use this information to help identify the environment as an office space or a warehouse; if the camera detects natural light within a certain wavelength range and / or detects motorized vehicles, the system may use this information to help identify the environment as outdoors; and so on.

[0156] In various embodiments of this implementation, the map information may include map data regarding the physical relationship between two or more controllable objects in an environment. For example, the map data may include one or more established center points, and the physical relationship between the two or more controllable objects in the environment may be determined from the established center points. The map data may include 3D panoramic data of the objects in the environment. In one example, the 3D panoramic data may include a partial view of the environment. In another example, the 3D panoramic data may include a representation of the environment along a single rotation axis or along multiple different rotation axes.

[0157] In various embodiments of this implementation, the environment may be an XR environment including virtual objects and real-world objects. The at least one computer processor may be programmed to: determine a reference object in the environment based on information about the environment captured by at least one camera; and determine position information of the virtual object and position information of the real-world object based on map information. The map information may include position information of at least one controllable object relative to the reference object.

[0158] In various embodiments of the implementation, the neuromuscular activity discerned from the neuromuscular signals sensed by the plurality of neuromuscular sensors is caused by a user performing at least one gesture relative to at least one controllable object while in the environment. The at least one gesture may include any one of the following or any combination thereof: the user moving at least one finger relative to the at least one controllable object (e.g., the user moving at least one finger up or down relative to the at least one controllable object); the user moving a wrist relative to the at least one controllable object (e.g., the user tilting a wrist up or down relative to the at least one controllable object); the user moving an arm relative to the at least one controllable object (e.g., the user moving an arm up or down relative to the at least one controllable object); and the user performing a pinching motion relative to the at least one controllable object using two or more fingers.

[0159] In various embodiments of the present implementation, the at least one controllable object may include a plurality of controllable objects. The at least one gesture may include a gesture relative to one of the plurality of controllable objects. The at least one computer processor may be programmed to control each of the plurality of controllable objects to change from a first state to a second state in response to neuromuscular activity discerned from neuromuscular signals sensed by the plurality of neuromuscular sensors.

[0160] In various embodiments of the present implementation, the environment may be any one or any combination of the following: one or more rooms in a home; one or more rooms in a business; one or more floors of a multi-story building; and an outdoor area.

[0161] In various embodiments of the present implementation, the at least one controllable object includes any one or any combination of the following: a light, a display device, an electronic game console, a curtain, a sound system, a lock, and a food or beverage preparation device.

[0162] Figure 13 A flow chart of a processing flow 1300 for an embodiment of this implementation is shown. At S1302, an image / video of an environment is captured by a camera. For example, the image / video may be captured by a camera attached to a strap worn around a user's arm. At S1304, the image / video is processed to determine whether a reference object can be identified. If not, the processing flow 1300 returns to S1302. If so, at S1306, a 3D map of the environment is accessed based on the identified reference object. Additionally, at S1306, one or more control interfaces are identified for one or more smart devices in the environment. For example, the control interface may be linked to the 3D map (e.g., see Figure 12). At S1308, the neuromuscular signals of the user are continuously sensed by the neuromuscular sensor, and the image / video continues to be captured. For example, the neuromuscular sensor can be attached to a strap to which the camera is attached. Processing flow 1300 proceeds to S1310, where it is determined whether the neuromuscular signals include one or more signals corresponding to the first neuromuscular activity. If so, processing flow 1300 proceeds to S1312; if not, processing flow 1300 returns to S1308. For example, the first neuromuscular activity can be the user pointing with a finger. At S1312, the 3D map is used to identify the smart device corresponding to the first neuromuscular activity, and the control interface for the identified smart device is accessed (for example, via a link to the 3D map). The field of view of the 3D camera can be aligned with the direction of the user's arm so that the user's finger can be captured in the image / video to facilitate the identification of the smart device. For example, the smart device can be a curtain (for example, Figure 9 908 in the ).

[0163] At S1314, it is determined whether the neuromuscular signal includes one or more signals corresponding to a second neuromuscular activity. If not, the process flow 1300 proceeds to S1318. If yes, the process flow 1300 proceeds to S1316, at which the identified smart device is controlled according to the second neuromuscular activity. Continuing with the previous example of the curtains, if the second neuromuscular activity is the user pinching their thumb and index finger together, the curtains can be controlled to close by an electronic mechanism activated via a control interface linked to the 3D map. At S1318, it is determined whether the neuromuscular signal includes one or more signals corresponding to a third neuromuscular activity. If not, the process flow 1300 returns to S1308. If yes, the process flow 1300 proceeds to S1320, at which the sensing and processing of the neuromuscular signal stops, and the capture of the image stops.

[0164] In some embodiments of this implementation, while the user is in an environment, the user can wear the wearable system 800, which can continuously stream image data, periodically stream image data (e.g., A seconds of imaging "on" followed by B seconds of imaging "off," etc., repeating sequence), or occasionally stream image data (e.g., C seconds of "on" followed by "off" until a neuromuscular event is detected). In these embodiments, the image data can be used to continuously, periodically, or occasionally "localize" or determine the user's position and / or orientation in the environment. In the case of occasionally streaming image data, when the user, for example, points with his or her finger in the environment, the C seconds of image data can be stored (e.g., in a cache) and updated with new C seconds of image data; thus, when the user performs an activity, processing of the new C seconds of image data and neuromuscular signals can be performed together, but otherwise processing is deferred to conserve processing resources. In this regard, to conserve processing resources, differencing techniques can be used to store, for example, the difference between multiple image frames, rather than storing all of them. For example, if a user moves slowly in an environment, and the image frames show slow changes in a wall as the user moves, data corresponding to the changes may be stored rather than the entirety of the image frames.

[0165] In some embodiments of this implementation, the wearable system 800 worn by the user may include an IMU sensor that helps locate the user in the environment (i.e., determine the user's position and orientation). For example, data from the IMU sensor can improve the positioning process by taking into account human arm limitations (discussed above) to give a more accurate determination of the user's arm orientation (e.g., angle). By using known human arm limitations to eliminate impossible arm positioning as part of the positioning process, the user's positioning can be determined more quickly. For example, a typical human arm can rotate through an arc of, for example, ±135° (based on a maximum possible arc of ±180° corresponding to a full circle). The limitations on the user's arm positioning / orientation will eliminate the possibility of the user pointing to an object that requires the user's arm to rotate greater than ±135°, thereby eliminating unnecessary calculations.

[0166] Implementation C - Hardware, including programming hardware

[0167] According to some embodiments of implementations of the present technology, an electronic device is provided. The device may include: a wearable carrier; a plurality of neuromuscular sensors attached to the carrier; a camera system; and at least one computer processor configured to electronically communicate with the plurality of neuromuscular sensors and the camera system. The device may also include a communication interface configured to transmit signals between the plurality of neuromuscular sensors, the camera system, and the at least one computer processor. The communication interface may include any one of the following or any combination thereof: wiring that directly interconnects the plurality of neuromuscular sensors and the camera system; wiring that directly interconnects the plurality of neuromuscular sensors and the at least one computer processor; wiring that directly interconnects the camera system and the at least one computer processor; a wired communication bus that interconnects the plurality of neuromuscular sensors, the camera system, and the at least one computer processor; a wireless signal transmitter; and a wireless signal receiver.

[0168] In various embodiments of this implementation, at least one processor may be attached to a carrier and may be configured to access the memory device to retrieve information and to store information.The memory device may be attached to the carrier.

[0169] In an embodiment of this implementation, the camera system may include at least two optical paths. For example, the camera system may include a stereo camera.

[0170] In multiple embodiments of this implementation, the camera system may include: an imaging portion and a depth determination portion. The imaging portion may include any one of the following or any combination thereof: a still image camera, an RGB camera, a panoramic camera, and a video camera. The imaging portion may be equipped with various types of lenses and / or filters. For example, the imaging portion may be equipped with a wide-angle or fisheye lens, which enables the imaging portion to capture a larger area of ​​the environment, which in some cases is used to accelerate the tracking of objects in the environment. The depth determination portion may include a light beam emitter and a receiver. For example, the light beam emitter and the receiver may be an infrared light beam emitter and a receiver. At least one computer processor may include a controller configured to control the imaging portion and the depth determination portion to capture data simultaneously.

[0171] In multiple embodiments of this implementation, at least one computer processor can be programmed to: receive an image signal from the imaging portion; receive a depth signal from the depth determination portion; generate correlation data that associates at least a portion of the image signal with at least a portion of the depth signal; and cause a memory device to store the correlation data.

[0172] In various embodiments of this implementation, the carrier can be configured to be worn by the user. In one example, the carrier can be configured to be worn on the user's hand (e.g., a glove). In another example, the carrier can be an armband configured to be worn on the user's arm (e.g., an elastic band, an adjustable band). In this example, the armband can be sized to surround the user's wrist or forearm, and the plurality of neuromuscular sensors are circumferentially arranged on the armband. The plurality of neuromuscular sensors can include an EMG sensor.

[0173] In multiple embodiments of this implementation, the camera system can be circumferentially arranged on the armband along with the multiple neuromuscular sensors. For example, the camera system can be radially arranged on the armband outside of one or more of the multiple neuromuscular sensors. The camera system can be arranged on the armband so as to be movable back and forth in a vertical orientation and an axial orientation, wherein the camera system points radially outward from the armband when the armband is worn on the user's arm, and in an axial orientation, wherein the camera system points axially in a direction parallel to the central axis of the armband when the armband is worn on the user's arm. The camera system can be attached to a hinge configured to pivot the camera system from a vertical orientation to an axial orientation, so that the camera system can capture image information in a vertical orientation, an axial orientation, or an orientation between the vertical orientation and the axial orientation.

[0174] In various embodiments of the present implementation, the apparatus may further include a second carrier configured to be worn on the user's head. The camera system may be attached to the second carrier.

[0175] In various embodiments of the present implementation, the apparatus may further include an auxiliary device attached to the wearable carrier or the second wearable carrier. The auxiliary device may include any one or any combination of the following: an IMU, a GPS, a radiation detector, a heart rate monitor, a moisture (e.g., perspiration) detector, etc.

[0176] Implementation D — Interaction via neuromuscular activity

[0177] According to some embodiments of implementations of the present technology, a computerized system for performing interactions via neuromuscular activity is provided. The system may include: a plurality of neuromuscular sensors, a camera system, and at least one computer processor. The plurality of neuromuscular sensors that can be configured to sense neuromuscular signals from a user can be arranged on at least one wearable device worn by the user to obtain the neuromuscular signals. The camera system that can be configured to capture information about an environment of interest to the user may include an imaging portion and a depth determination portion. The at least one computer processor may be programmed to: receive captured information from the camera system and receive neuromuscular signals from the plurality of neuromuscular sensors; discern an environment from the captured information; access control information associated with the environment discerned from the captured information, the control information including information for performing at least one function associated with the environment; and, in response to the neuromuscular activity discerned from the neuromuscular signals, cause the at least one function to be performed.

[0178] In various embodiments of this implementation, the environment may be an outdoor area, and the control information may include information for performing an outdoor function. The outdoor function may be a transportation-related function. In one example, the transportation-related function may be starting a car. In another example, the transportation-related function may be causing a pickup request to be sent in the outdoor area. The at least one computer processor may be programmed to transmit the request to a transportation service provider via an internet connection.

[0179] In various embodiments of this implementation, at least one computer processor may be programmed to determine the presence of a person in the environment based on the captured information. For example, at least one computer processor may be programmed to determine the presence of a person in the environment based on one or any combination of the following: a discerned general shape of the person, the presence of at least one limb of the person, discerned facial features of the person, discerned movement of the person, and an object carried by the person. The at least one computer processor may also be programmed to determine the identity of the person based on the discerned facial features of the person if the person is determined to be in the environment.

[0180] In various embodiments of this implementation, the system may further include a communication interface configured to transmit electronic signals to and receive electronic signals from an external device. The external device may be any one or any combination of the following: a smart device, a smartphone, a haptic device, a gaming console, and a display device. In some embodiments, the at least one computer processor may be programmed to determine the identity of a person in the environment based on the identified facial features of the person; access an interactive control application based on the person's identity; and, using the interactive control application, cause any one or any combination of the following to be performed: transmitting a personal message to the person's smartphone, executing a game move in an electronic gaming console played by the person, activating a haptic device worn by the person, displaying an image on a display device viewable by the person in the environment, and allowing the person to control a function of the environment. The at least one computer processor may be programmed to cause the person granted permission to control any one or any combination of the following: an electronic gaming console, a display device, a user's smartphone, and a haptic device worn by the user.

[0181] In various embodiments of this implementation, at least one computer processor may be programmed to utilize an interactive control application to generate an XR environment in which a user and a person can interact. In one example, the at least one computer processor may be programmed to enable a user to interact with a person in the XR environment based on neuromuscular activity discerned from neuromuscular signals sensed by a plurality of neuromuscular sensors. In another example, the at least one computer processor may be programmed to utilize the interactive control application to: turn on an electronic device operable by the user and a person, and control at least one operation of the electronic device in response to one or more neuromuscular activities discerned from the neuromuscular signals sensed by the plurality of neuromuscular sensors. For example, the electronic device may be a game console playable by the user and a person, and in response to one or more neuromuscular activities discerned from the neuromuscular signals, the at least one computer processor controls any one or any combination of the following: game player movement, game effects, and game settings.

[0182] Figure 14A and Figure 14BA flowchart of a process flow 1400 for an embodiment of this implementation is shown. At S1402, a camera captures an image / video of an environment. For example, the image / video may be captured by a camera attached to a strap worn around a user's arm. At S1404, the image / video is processed to determine whether a reference object can be identified. If not, process flow 1400 returns to S1402. If so, at S1406, a 3D map of the environment is accessed based on the identified reference object. Furthermore, at S1406, one or more control interfaces are identified for the environment. For example, the control interfaces may be used to link communication functions of the environment to the 3D map. At S1408, the user's neuromuscular signals are continuously sensed by a neuromuscular sensor, and image / video capture continues. For example, the neuromuscular sensor may be attached to the strap to which the camera is attached. Process flow 1400 proceeds to S1410, where it is determined whether a person is detected in the environment. For example, detection may be determined by determining that the shape of the object in the image / video resembles a person. If a person is detected, the process flow 1400 proceeds to S1412; if not, the process flow 1400 returns to S1408.

[0183] At S1412, processing is performed to identify the person. For example, if the person carries a smartphone or other electronic device (e.g., another neuromuscular armband or wristband), the signal from the smartphone or other electronic device may be detected by an electronic circuit (e.g., NFC circuit, RFID circuit, etc.) attached to the band to which the camera is attached. In another example, facial detection may be used to identify the person from the image / video. In yet another example, the person may wear a wearable system (e.g., Figure 8A The wearable system 800 in FIG. 1 may provide an identification signal that can be used to identify a person. At step S1414, if a person cannot be identified, process flow 1400 returns to step S1408. If a person is identified at step S1414, process flow 1400 proceeds to step S1416, where one or more control interfaces associated with the identified person are identified or accessed for use.

[0184] At S1418, it is determined whether the neuromuscular signals include one or more signals corresponding to neuromuscular activity for an action in the environment or an interaction with an identified person. If so, the process 1400 proceeds to S1422; if not, the process 1400 returns to S1418 via S1420, where it continues to sense and process the user's neuromuscular signals to determine whether there is neuromuscular activity for an action in the environment or an interaction with an identified person.

[0185] At S1422, a control interface associated with the identified person and / or a control interface associated with the environment is used to interact with the identified person (e.g., send a text message to the identified person's smartphone) and / or to control actions in the environment (e.g., activate an XR function in the environment). At S1424, a determination is made as to whether the neuromuscular signals include one or more signals corresponding to neuromuscular activity to exit control operations and interactions in the environment. If not, process flow 1400 proceeds to S1418 via S1420. If yes, process flow 1400 proceeds to S1426, where sensing and processing of the neuromuscular signals ceases, and image / video capture ceases. If the system interacts with the identified person, such interaction may include any one or any combination of the following: sending a discrete text or email message to the identified person's mobile phone or computer, synchronizing one or more smart devices between the user and the identified person, sending a message or communication in the XR environment that the identified person can view in the XR environment, and the like.

[0186] In some embodiments, two or more users can interact with the XR environment. Each user (e.g., user A and user B) can interact with the XR system through a wearable device (e.g., Figure 8A In this case, facial recognition is not required because each user's wearable device will be registered to a different account in the XR system, so that each wearable device will uniquely identify the user to the XR system. Therefore, each user can interact with objects in the XR environment independently of another user, and can also interact with another user in the XR environment. In addition, each user can be associated with a different 3D map for the same XR environment (for example, user A can be a child and can only have control over the lights in the XR environment of user A's 3D map, while user B can have control over the TV and video game system in the XR environment of user B's 3D map). In an embodiment, when multiple users are in the same XR environment at the same time, the XR system can enable all users to share control of each other's controllable objects (for example, user A and user B can both perform control operations on the video game system). In this regard, when user A and user B are in the same XR environment, user A's 3D map can be merged or spliced ​​together with user B's 3D map.

[0187] In a cloud-based embodiment, an environment may be associated with multiple users, and each user may be equipped with a wearable band (e.g., Figure 8AWearable system 800 in (a). For example, the environment may be a workplace having multiple distinct areas, and multiple users may be employees of the workplace. Employee A and employee C may be associated with map A1 for area A of the environment and map C for area C. Employee B and employee D may be associated with map A2 for area A of the environment and map B for area B. Map A1, map A2, map B, and map C may be stored remotely in a cloud facility that may be controlled by a workplace administrator. Thus, the workplace administrator may authorize a first group of employees to have permission to control a first group of objects within area A via map A1, and may authorize a second group of employees to have permission to control a second group of objects within the same area A via map A2.

[0188] Workplace managers can control the authorization details of each employee via the employee's wearable band, and can change the authorization of an employee via the employee's wearable band. Each wearable band can be associated with an identification code, which can be used to associate the wearable band with one or more 3D maps of an area of ​​the environment. For example, the environment can be a warehouse with multiple inventory rooms, and map A1 can allow one group of employees to operate heavy machinery in area A, while map A2 can allow another group of employees to operate light machinery in area A. It will be understood that some employees can be associated with maps A1 and A2, and therefore be allowed to operate both light machinery and heavy machinery. It will be understood that workplace managers can use 3D maps about areas of the environment to set various other types of controls for areas of the environment, and that the above description is merely illustrative of one example.

[0189] When in the environment, each wearable band can emit an identification signal (e.g., using RFID technology, NFC technology, GPS technology, etc.). Such signals can be used to determine which employee is in the environment, and roughly where the employee is in the environment (e.g., which area of ​​the environment); the positioning technology described above (e.g., using video streams) can be used to determine the user's more precise location (e.g., the employee's location within the area of ​​the environment). In some embodiments, such signals can be used to access one or more 3D maps, which can enable the employee corresponding to the signal to control objects in the environment or otherwise interact with the environment (e.g., enabling a first employee in the environment to interact with a second employee in the environment via a neuromuscular action indicating a desire to transfer control authority from the first employee to the second employee; enabling the first employee to send a tactile warning signal to the second employee, etc.).

[0190] It will be appreciated that cloud-based storage of 3D maps and other information available to managers can enable centralized oversight of a workplace environment that may include regions across multiple cities and / or multiple states and / or multiple countries. For example, managers can control employees by controlling the authorizations or permissions granted to each employee, which can differ from employee to employee based on each employee's wearable band and the 3D map associated with each employee's wearable band.

[0191] Example Embodiments

[0192] Example 1: A computerized system for remotely controlling a device may include a plurality of neuromuscular sensors configured to sense neuromuscular signals from a user, wherein the plurality of neuromuscular sensors are arranged on at least one wearable device configured to be worn by the user to obtain the neuromuscular signals. The system may also include at least one camera capable of capturing information about an environment and at least one computer processor programmed to: access map information of the environment based on the information about the environment captured by the at least one camera, the map information including information for controlling at least one controllable object in the environment, and control the at least one controllable object from a first state to a second state in response to neuromuscular activity discerned from the neuromuscular signals sensed by the plurality of neuromuscular sensors.

[0193] Example 2: The system of Example 1, wherein: map information of the environment is stored in a memory, and the at least one computer processor retrieves the map information from the memory based on information discerned from information about the environment captured by the at least one camera.

[0194] Example 3: A system according to Example 1 or 2, wherein the discerned information is visible in the environment and includes at least one of the following: a QR code, a graphic symbol, an alphanumeric text string, a 3D object with a specific shape, or a physical relationship between at least two objects.

[0195] Example 4: The system of any of Examples 1-3, wherein the discerned information includes a reference object of the environment.

[0196] Example 5: The system of any of Examples 1-4, wherein the map information includes map data representing a physical relationship between two or more controllable objects in the environment.

[0197] Example 6: The system of any one of Examples 1-5, wherein the map data is 3D panoramic data of objects in the environment, the objects including at least one controllable object.

[0198] Example 7: A system according to any of Examples 1-6, wherein the 3D panoramic data includes a 360° representation of the environment.

[0199] Example 8: The system of any of Examples 1-7, wherein the 3D panoramic data comprises data relative to a single rotational axis.

[0200] Example 9: The system of any of Examples 1-8, wherein the 3D panoramic data comprises a representation of a local view of an environment.

[0201] Example 10: A system according to any of Examples 1-9, wherein the map information includes a plurality of established center points, and wherein a physical relationship between two or more controllable objects in the environment is determined from the established center points.

[0202] Example 11: A system according to any of Examples 1-10, wherein the environment is an extended reality (XR) environment including virtual objects and real-world objects, wherein the at least one computer processor is programmed to: determine position information of the virtual objects and position information of the real-world objects based on the map information, and determine a reference object in the environment based on information about the environment captured by the at least one camera, and wherein the map information includes position information of the at least one controllable object relative to the reference object.

[0203] Example 12: A system according to any of Examples 1-11, wherein the neuromuscular activity discerned from the neuromuscular signals sensed by the plurality of neuromuscular sensors is caused by the user performing at least one gesture relative to the at least controllable object while the user is in the environment.

[0204] Example 13: The system of any of Examples 1-12, wherein the at least one gesture includes any one of the following or any combination thereof: the user moves at least one finger relative to the at least one controllable object; the user moves a wrist relative to the at least one controllable object; the user moves an arm relative to the at least one controllable object; the user applies a force but does not move relative to the at least one controllable object; and the user activates a motor unit but does not move relative to the at least one controllable object and has no force relative to the at least one controllable object.

[0205] Example 14: A system according to any of Examples 1-13, wherein the neuromuscular activity discerned from the neuromuscular signals sensed by the plurality of neuromuscular sensors is motor unit activation potential (MUAP) activity performed by the user relative to the at least controllable object when the user is in the environment.

[0206] Example 15: A system according to any of Examples 1-14, wherein the at least one gesture includes at least one of: the user using two or more fingers to perform a pinching motion relative to the at least one controllable object, the user tilting the wrist up or down relative to the at least one controllable object, the user moving the arm up or down relative to the at least one controllable object, or the user moving at least one finger up or down relative to the at least one controllable object.

[0207] Example 16: A system according to any one of Examples 1-15, wherein: the at least one controllable object includes a plurality of controllable objects, the at least one posture includes a posture relative to a controllable object of the plurality of controllable objects, and the at least one computer processor is programmed to control each of the plurality of controllable objects to change from a first state to a second state in response to the control signal.

[0208] Example 17: The system of any of Examples 1-16, wherein the environment is at least one of: a room in a home, a room in a business, a floor of a multi-story building, or an outdoor area.

[0209] Example 18: A system according to any of Examples 1-17, wherein the at least one controllable object includes any one or any combination of the following: a display device, a light, an electronic game console, a curtain, a sound system, a lock, and food or beverage preparation equipment.

[0210] Example 19: A computer-implemented method comprising: activating a plurality of neuromuscular sensors configured to sense neuromuscular signals from a user, the plurality of neuromuscular sensors being arranged on at least one wearable device configured to be worn by the user to obtain the neuromuscular signals; activating at least one camera capable of capturing information about an environment; accessing map information of the environment based on the information about the environment captured by the at least one camera, the map information including information for controlling at least one controllable object in the environment; and controlling the at least one controllable object from a first state to a second state in response to neuromuscular activity discerned from the neuromuscular signals sensed by the plurality of neuromuscular sensors.

[0211] Example 20: A wearable electronic device comprising: a plurality of neuromuscular sensors configured to sense neuromuscular signals from a user, the plurality of neuromuscular sensors being arranged on at least one wearable device configured to be worn by the user to obtain the neuromuscular signals; at least one camera capable of capturing information about an environment; and at least one computer processor programmed to access map information of the environment based on the information about the environment captured by the at least one camera, the map information including information for controlling at least one controllable object in the environment, and to control the at least one controllable object from a first state to a second state in response to neuromuscular activity discerned from the neuromuscular signals sensed by the plurality of neuromuscular sensors.

[0212] Embodiments of the present disclosure may include or be implemented in combination with various types of artificial reality systems. Artificial reality is a form of reality that has been adjusted in some way before being presented to a user, which may include, for example, virtual reality, augmented reality, mixed reality, hybrid reality, or some combination and / or derivative thereof. Artificial reality content may include content that is entirely computer-generated or computer-generated content combined with captured (e.g., real-world) content. Artificial reality content may include video, audio, tactile feedback, or some combination thereof, any of which may be presented in a single channel or multiple channels (e.g., stereoscopic video that produces a three-dimensional (3D) effect to the viewer). In addition, in some embodiments, artificial reality may also be associated with applications, products, accessories, services, or some combination thereof, which are used, for example, to create content in artificial reality and / or to be used in artificial reality in other ways (e.g., to perform activities in artificial reality).

[0213] Artificial reality systems can be implemented in a variety of different form factors and configurations. Some artificial reality systems can be designed to operate without a near-eye display (NED). Other artificial reality systems can include an NED that also provides visibility into the real world (such as, for example, Figure 15 ), or visually immersing a user in an artificial reality (such as, for example, Figure 16 16). While some artificial reality devices may be standalone systems, other artificial reality devices may communicate and / or coordinate with external devices to provide an artificial reality experience to a user. Examples of such external devices include handheld controllers, mobile devices, desktop computers, devices worn by a user, devices worn by one or more other users, and / or any other suitable external system.

[0214] Go to Figure 15 , the augmented reality system 1500 may include an eyewear device 1502 having a frame 1510, wherein the frame 3210 is configured to hold a left display device 1515(A) and a right display device 1515(B) in front of the user's eyes. Display devices 1515(A) and 1515(B) may function together or independently to present an image or a series of images to the user. Although the augmented reality system 1500 includes two displays, embodiments of the present disclosure may be implemented in an augmented reality system having a single NED or more than two NEDs.

[0215] In some embodiments, augmented reality system 1500 may include one or more sensors, such as sensor 1540. Sensor 1540 may generate measurement signals in response to the movement of augmented reality system 1500 and may be located on substantially any portion of frame 1510. Sensor 1540 may represent one or more of a variety of different sensing mechanisms, such as a position sensor, an inertial measurement unit (IMU), a depth camera assembly, a structured light emitter and / or detector, or any combination thereof. In some embodiments, augmented reality system 1500 may include or exclude sensor 1540, or may include more than one sensor. In embodiments where sensor 1540 includes an IMU, the IMU may generate calibration data based on the measurement signals from sensor 1540. Examples of sensor 1540 may include, but are not limited to, accelerometers, gyroscopes, magnetometers, other suitable types of sensors that detect movement, sensors used for error correction of the IMU, or some combination thereof.

[0216] In some examples, augmented reality system 1500 may also include a microphone array having a plurality of acoustic transducers 1520(A)-1520(J) (collectively referred to as acoustic transducers 1520). Acoustic transducers 1520 may represent transducers that detect changes in air pressure caused by sound waves. Each acoustic transducer 1520 may be configured to detect sound and convert the detected sound into an electronic format (e.g., analog or digital format). Figure 15 The microphone array in can include, for example, ten acoustic transducers: 1520(A) and 1520(B), which can be designed to be placed within the user's respective ears; acoustic transducers 1520(C), 1520(D), 1520(E), 1520(F), 1520(G), and 1520(H), which can be located at different locations on the frame 1510; and / or acoustic transducers 1520(I) and 1520(J), which can be located on the respective neckbands 1505.

[0217] In some embodiments, one or more of acoustic transducers 1520(A)-(J) can be used as an output transducer (e.g., a speaker). For example, acoustic transducers 1520(A) and / or 1520(B) can be earbuds or any other suitable type of headphones or speakers.

[0218] The configuration of the acoustic transducers 1520 of the microphone array can vary. Figure 15 15. The frame 1510 is shown as having ten acoustic transducers 1520, but the number of acoustic transducers 1520 can be greater or less than ten. In some embodiments, using a greater number of acoustic transducers 1520 can increase the amount of audio information collected and / or the sensitivity and accuracy of the audio information. Conversely, using a lower number of acoustic transducers 1520 can reduce the computing power required by the associated controller 1550 to process the collected audio information. In addition, the position of each acoustic transducer 1520 of the microphone array can vary. For example, the position of the acoustic transducer 1520 can include a position defined on the user, defined coordinates on the frame 1510, an orientation associated with each acoustic transducer 1520, or some combination thereof.

[0219] Acoustic transducers 1520(A) and 1520(B) can be located at different parts of the user's ear, such as behind the pinna, behind the tragus, and / or inside the auricle or fossa. Alternatively, in addition to the acoustic transducer 1520 in the ear canal, additional acoustic transducers 1520 can be present on or around the ear. Positioning the acoustic transducer 1520 near the user's ear canal can enable the microphone array to collect information about how sound reaches the ear canal. By positioning at least two acoustic transducers 1520 on either side of the user's head (e.g., as binaural microphones), the augmented reality device 1500 can simulate binaural hearing and capture a 3D stereo sound field around the user's head. In some embodiments, acoustic transducers 1520(A) and 1520(B) may be connected to augmented reality system 1500 via a wired connection 1530, and in other embodiments, acoustic transducers 1520(A) and 1520(B) may be connected to augmented reality system 1500 via a wireless connection (e.g., a Bluetooth connection). In other embodiments, acoustic transducers 1520(A) and 1520(B) may not be used in conjunction with augmented reality system 1500 at all.

[0220] The acoustic transducers 1520 on the frame 1510 can be positioned in a variety of different ways, including along the length of the temples, across the bridge, above or below the display devices 1515(A) and 1515(B), or some combination thereof. The acoustic transducers 1520 can also be oriented so that the microphone array can detect sounds from a wide range of directions around the user wearing the augmented reality system 1500. In some embodiments, an optimization process can be performed during the manufacture of the augmented reality system 1500 to determine the relative positioning of each acoustic transducer 1520 in the microphone array.

[0221] In some examples, augmented reality system 1500 can include or be connected to an external device (e.g., a paired device), such as neckband 1505. Neckband 1505 generally represents any type or form of paired device. Therefore, the following discussion of neckband 1505 can also be applied to various other paired devices, such as charging cases, smart watches, smartphones, wristbands, other wearable devices, handheld controllers, tablet computers, laptop computers, other external computing devices, etc.

[0222] As shown, the neckband 1505 can be coupled to the eyeglass device 1502 via one or more connectors. The connectors can be wired or wireless and can include electrical and / or non-electrical (e.g., structural) components. In some cases, the eyeglass device 1502 and the neckband 1505 can operate independently without any wired or wireless connection between them. Figure 15 Components of the eyeglass device 1502 and neckband 1505 are illustrated in example locations on the eyeglass device 1502 and neckband 1505, but these components may be located elsewhere on the eyeglass device 1502 and / or neckband 1505 and / or distributed differently on the eyeglass device 1502 and / or neckband 1505. In some embodiments, components of the eyeglass device 1502 and neckband 1505 may be located on one or more additional peripheral devices that are paired with the eyeglass device 1502, the neckband 1505, or some combination thereof.

[0223] Pairing an external device (e.g., neckband 1505) with an augmented reality eyewear device can enable the eyewear device to achieve the form factor of a pair of glasses while still providing sufficient battery and computing power for expanded functionality. Some or all of the battery power, computing resources, and / or other features of the augmented reality system 1500 can be provided by the paired device or shared between the paired device and the eyewear device, thereby reducing the weight, heat profile, and form factor of the eyewear device overall while still maintaining the desired functionality. For example, the neckband 1505 can allow components that would otherwise be included on the eyewear device to be included in the neckband 1505, as the user can tolerate a heavier weight load on their shoulders than on their head. The neckband 1505 can also have a larger surface area over which to spread and dissipate heat into the surrounding environment. Thus, the neckband 1505 can allow for a larger battery and computing capacity than would otherwise be feasible on a standalone eyewear device. Because the weight carried in the neckband 1505 can be less intrusive to the user than the weight carried in the eyeglass device 1502, the user can tolerate wearing a lighter eyeglass device and carry or wear the paired device for longer periods of time than the user would tolerate wearing a heavier standalone eyeglass device, thereby enabling the user to more fully integrate the artificial reality environment into their daily activities.

[0224] The neckband 1505 can be communicatively coupled to the eyewear device 1502 and / or other devices. These other devices can provide certain functions (e.g., tracking, positioning, depth mapping, processing, storage, etc.) to the augmented reality system 1500. Figure 15 In an embodiment, the neckband 1505 may include two acoustic transducers (e.g., 1520(I) and 1520(J)) that are part of a microphone array (or potentially form their own microphone subarray). The neckband 1505 may also include a controller 1525 and a power supply 1535.

[0225] The acoustic transducers 1520(I) and 1520(J) of the neckband 1505 may be configured to detect sound and convert the detected sound into an electronic format (analog or digital). Figure 15In some embodiments, acoustic transducers 1520(I) and 1520(J) can be positioned on neckband 1505, thereby increasing the distance between neckband acoustic transducers 1520(I) and 1520(J) and other acoustic transducers 1520 positioned on eyeglass device 1502. In some cases, increasing the distance between acoustic transducers 1520 of a microphone array can improve the accuracy of beamforming performed via the microphone array. For example, if acoustic transducers 1520(C) and 1520(D) detect a sound, and the distance between acoustic transducers 1520(C) and 1520(D) is greater than, for example, the distance between acoustic transducers 1520(D) and 1520(E), the source location of the detected sound may be determined to be more accurate than if the sound were detected by acoustic transducers 1520(D) and 1520(E).

[0226] The controller 1525 of the neckband 1505 can process information generated by sensors on the neckband 1505 and / or by the augmented reality system 1500. For example, the controller 1525 can process information from the microphone array describing sounds detected by the microphone array. For each detected sound, the controller 1525 can perform a direction of arrival (DOA) estimate to estimate the direction from which the detected sound arrived at the microphone array. When the microphone array detects a sound, the controller 1525 can populate the audio data set with this information. In embodiments where the augmented reality system 1500 includes an inertial measurement unit, the controller 1525 can calculate all inertial and spatial calculations from the IMU located on the eyewear device 1502. A connector can transfer information between the augmented reality system 1500 and the neckband 1505, and between the augmented reality system 1500 and the controller 1525. This information can be in the form of optical data, electrical data, wireless data, or any other transmittable data form. Moving the processing of information generated by the augmented reality system 1500 to the neckband 1502 can reduce weight and heat in the eyewear device 1502, making it more comfortable for the user.

[0227] A power supply 1535 in the neckband 1505 can provide power to the eyewear device 1502 and / or the neckband 1505. The power supply 1535 can include, but is not limited to, a lithium-ion battery, a lithium-polymer battery, a primary lithium battery, an alkaline battery, or any other form of power storage device. In some cases, the power supply 1535 can be a wired power source. Including the power supply 1535 on the neckband 1505 rather than on the eyewear device 1502 can help better distribute the weight and heat generated by the power supply 1535.

[0228] As mentioned, some artificial reality systems can essentially replace one or more of the user's sensory perceptions of the real world with a virtual experience, rather than mixing the artificial reality with the actual reality. An example of this type of system is a head-mounted display system, e.g. Figure 16The virtual reality system 1600 in FIG. 1 mainly or completely covers the user's field of view. The virtual reality system 1600 may include a front rigid body 1602 and a strap 1604 shaped to surround the user's head. The virtual reality system 1600 may also include output audio transducers 1606 (A) and 1606 (B). In addition, although Figure 16 Not shown, but front rigid body 1602 may include one or more electronic components, including one or more electronic displays, one or more inertial measurement units (IMUs), one or more tracking transmitters or detectors, and / or any other suitable device or system for creating an artificial reality experience.

[0229] Artificial reality systems may include various types of visual feedback mechanisms. For example, the display devices in the augmented reality system 1500 and / or virtual reality system 1600 may include one or more liquid crystal displays (LCDs), light emitting diode (LED) displays, organic LED (OLED) displays, digital light projection (DLP) microdisplays, liquid crystal on silicon (LCoS) microdisplays, and / or any other suitable type of display screen. These artificial reality systems may include a single display screen for both eyes, or a display screen may be provided for each eye, which may provide additional flexibility for zoom adjustment or for correcting the user's refractive error. Some of these artificial reality systems may also include an optical subsystem having one or more lenses (e.g., traditional concave or convex lenses, Fresnel lenses, adjustable liquid lenses, etc.) through which the user can view the display screen. These optical subsystems may be used for a variety of purposes, including collimation (e.g., making an object appear to be farther away than its physical distance), magnification (e.g., making an object appear larger than its actual size), and / or light transmission (e.g., transmitting light to the viewer's eyes). These optical subsystems can be used in non-pupil-forming architectures (e.g., a single-lens configuration that directly collimates light but results in so-called pincushion distortion) and / or pupil-forming architectures (e.g., a multi-lens configuration that produces so-called barrel distortion to offset pincushion distortion).

[0230] In addition to or instead of using a display screen, some artificial reality systems described herein may include one or more projection systems. For example, a display device in the augmented reality system 1500 and / or virtual reality system 1600 may include a micro-LED projector that projects light into the display device (using, for example, a waveguide), such as a transparent combination lens that allows ambient light to pass through. The display device can refract the projected light toward the user's pupil and can enable the user to view both the artificial reality content and the real world simultaneously. The display device can use any of a variety of different optical components to achieve this, including waveguide components (e.g., holographic, planar, diffractive, polarizing and / or reflective waveguide elements), light manipulation surfaces and elements (such as diffractive, reflective and refractive elements and gratings), coupling elements, etc. The artificial reality system can also be configured to have any other suitable type or form of image projection system, such as a retinal projector used in a virtual retinal display.

[0231] The artificial reality systems described herein may also include various types of computer vision components and subsystems. For example, the augmented reality system 1500 and / or the virtual reality system 1600 may include one or more optical sensors, such as two-dimensional (2D) or 3D cameras, structured light emitters and detectors, time-of-flight depth sensors, single-beam or scanning laser rangefinders, 3D LiDAR sensors, and / or any other suitable type or form of optical sensor. The artificial reality system may process data from one or more of these sensors to identify the user's location, map the real world, provide the user with context about their real-world surroundings, and / or perform a variety of other functions.

[0232] The artificial reality systems described herein may also include one or more input and / or output audio transducers. The output audio transducer may include a voice coil speaker, a ribbon speaker, an electrostatic speaker, a piezoelectric speaker, a bone conduction transducer, a cartilage conduction transducer, an auricle vibration transducer, and / or any other suitable type or form of audio transducer. Similarly, the input audio transducer may include a condenser microphone, a dynamic microphone, a ribbon microphone, and / or any other type or form of input transducer. In some embodiments, a single transducer may be used for both audio input and audio output.

[0233] In some embodiments, the artificial reality system described herein may also include a tactile (i.e., haptic) feedback system that can be incorporated into headgear, gloves, bodysuits, handheld controllers, environmental equipment (e.g., chairs, floor mats, etc.), and / or any other type of device or system. The tactile feedback system can provide various types of skin feedback, including vibration, force, traction, texture, and / or temperature. The tactile feedback system can also provide various types of kinesthetic feedback, such as motion and compliance. Tactile feedback can be implemented using motors, piezoelectric actuators, fluidic systems, and / or various other types of feedback mechanisms. The tactile feedback system can be implemented independently of other artificial reality devices, within other artificial reality devices, and / or in combination with other artificial reality devices.

[0234] By providing tactile sensations, audible content, and / or visual content, artificial reality systems can create entire virtual experiences or enhance the user's real-world experience in a variety of contexts and environments. For example, an artificial reality system can help or expand a user's perception, memory, or cognition within a specific environment. Some systems can enhance the user's interaction with other people in the real world, or can enable a more immersive interaction between the user and other people in the virtual world. Artificial reality systems can also be used for educational purposes (e.g., for teaching or training in schools, hospitals, government organizations, military organizations, commercial enterprises, etc.), entertainment purposes (e.g., for playing video games, listening to music, watching video content, etc.), and / or barrier-free purposes (e.g., as hearing aids, visual aids, etc.). The embodiments disclosed herein can implement or enhance the user's artificial reality experience in one or more of these contexts and environments and / or in other contexts and environments.

[0235] As described above, the augmented reality systems 1500 and 1600 can be used with various other types of devices to provide a more compelling artificial reality experience. These devices can be tactile interfaces with transducers that provide tactile feedback and / or collect tactile information about the user's interaction with the environment. The artificial reality systems disclosed herein can include various types of tactile interfaces that detect or convey various types of tactile information, including tactile feedback (e.g., feedback detected by the user through nerves in the skin, which may also be referred to as skin feedback) and / or kinesthetic feedback (e.g., feedback detected by the user through receptors located in muscles, joints, and / or tendons).

[0236] Haptic feedback can be provided by interfaces located within the user's environment (e.g., a chair, table, floor, etc.) and / or interfaces on items that the user can wear or carry (e.g., gloves, wristbands, etc.). As an example, Figure 17A vibrotactile system 1700 is shown in the form of a wearable glove (haptic device 1710) and a wristband (haptic device 1720). Haptic device 1710 and haptic device 1720 are shown as examples of wearable devices that include a flexible, wearable textile material 1730 that is shaped and configured to be positioned against a user's hand and wrist, respectively. The present disclosure also includes a vibrotactile system that can be shaped and configured to be positioned against other human body parts (e.g., a finger, arm, head, torso, foot, or leg). By way of example and not limitation, the vibrotactile system according to various embodiments of the present disclosure can also be in the form of a glove, a headband, an armband, a sleeve, a hood, a sock, a shirt, or pants, among other possibilities. In some examples, the term "textile" can include any flexible, wearable material, including woven fabrics, non-woven fabrics, leather, cloth, flexible polymer materials, composite materials, and the like.

[0237] One or more vibrotactile devices 1740 can be at least partially located within one or more corresponding pockets formed in the textile material 1730 of the vibrotactile system 1700. The vibrotactile devices 1740 can be positioned to provide a vibrotactile sensation (e.g., tactile feedback) to a user of the vibrotactile system 1700. For example, the vibrotactile devices 1740 can be positioned against a user's finger, thumb, or wrist, such as Figure 17 In some examples, vibrotactile device 1740 can be flexible enough to conform to or bend with a corresponding body part of a user.

[0238] A power source 1750 (e.g., a battery) for applying voltage to the vibrotactile device 1740 to activate it can be electrically coupled to the vibrotactile device 1740, such as via a wire 1752. In some examples, each vibrotactile device 1740 can be independently electrically coupled to the power source 1750 for individual activation. In some embodiments, a processor 1760 can be operably coupled to the power source 1750 and configured (e.g., programmed) to control the activation of the vibrotactile device 1740.

[0239] The vibrotactile system 1700 can be implemented in a variety of ways. In some examples, the vibrotactile system 1700 can be an independent system having integrated subsystems and components that operate independently of other devices and systems. As another example, the vibrotactile system 1700 can be configured to interact with another device or system 1770. For example, in some examples, the vibrotactile system 1700 can include a communication interface 1780 for receiving signals and / or sending signals to the other device or system 1770. The other device or system 1770 can be a mobile device, a game console, an artificial reality (e.g., virtual reality, augmented reality, mixed reality) device, a personal computer, a tablet computer, a network device (e.g., a modem, a router, etc.), a handheld controller, etc. The communication interface 1780 can realize communication between the vibrotactile system 1700 and the other device or system 1770 via a wireless (e.g., Wi-Fi, Bluetooth, cellular, radio, etc.) link or a wired link. If present, the communication interface 1780 may communicate with the processor 1760 , eg, providing signals to the processor 1760 to activate or deactivate one or more vibrotactile devices 1740 .

[0240] The vibrotactile system 1700 may optionally include other subsystems and components, such as a touch-sensitive pad 1790, a pressure sensor, a motion sensor, a position sensor, an illumination element, and / or a user interface element (e.g., an on / off button, a vibration control element, etc.). During use, the vibrotactile device 1740 may be configured to be activated for a variety of different reasons, such as in response to user interaction with a user interface element, a signal from a motion or position sensor, a signal from the touch-sensitive pad 1790, a signal from a pressure sensor, a signal from another device or system 1770, etc.

[0241] Although the power supply 1750, processor 1760 and communication interface 1780 are Figure 17 17. The haptic device 1720 is shown as being located in the haptic device 1720, but the present disclosure is not limited thereto. For example, one or more of the power supply 1750, the processor 1760, or the communication interface 1780 can be located within the haptic device 1710 or within another wearable textile.

[0242] Tactile wearable devices, such as Figure 17 What is shown and described can be implemented in various types of artificial reality systems and environments. Figure 18An example artificial reality environment 1800 is shown that includes a head-mounted virtual reality display and two haptic devices (i.e., gloves), and in other embodiments, any number and / or combination of these and other components may be included in the artificial reality system. For example, in some embodiments, there may be multiple head-mounted displays, each with an associated haptic device, each head-mounted display and each haptic device communicating with the same console, portable computing device, or other computing system.

[0243] Head mounted display 1802 generally represents any type or form of virtual reality system, such as Figure 16 Virtual reality system 1600 in the artificial reality system. Haptic device 1804 generally represents any type or form of wearable device worn by a user of the artificial reality system that provides tactile feedback to the user, giving the user the sensation of physically contacting a virtual object. In some embodiments, haptic device 1804 can provide tactile feedback by applying vibration, motion, and / or force to the user. For example, haptic device 1804 can restrict or increase the user's movement. As a specific example, haptic device 1804 can restrict the forward movement of a user's hand, giving the user the sensation of physical contact with a virtual wall. In this specific example, one or more actuators within the haptic device can achieve physical motion restriction by pumping fluid into an inflatable bladder in the haptic device. In some examples, the user can also use haptic device 1804 to send action requests to the console. Examples of action requests include, but are not limited to, requests to launch and / or terminate an application and / or requests to perform specific actions within an application.

[0244] Although haptic interfaces can be used with virtual reality systems, e.g. Figure 18 As shown, haptic interfaces can also be used with augmented reality systems, such as Figure 19 shown. Figure 19 19 is a perspective view of a user 1910 interacting with an augmented reality system 1900. In this example, the user 1910 may be wearing a pair of augmented reality glasses 1920, which may have one or more displays 1922 and be paired with a haptic device 1930. In this example, the haptic device 1930 may be a wristband that includes a plurality of strap elements 1932 and a tensioning mechanism 1934 connecting the strap elements 1932 to each other.

[0245] One or more band elements 1932 may include any type or form of actuator suitable for providing tactile feedback. For example, one or more of band elements 1932 may be configured to provide one or more of various types of skin feedback, including vibration, force, traction, texture, and / or temperature. To provide such feedback, band elements 1932 may include one or more of various types of actuators. In one example, each of band elements 1932 may include a vibrotactile (e.g., a vibrotactile actuator) that is configured to vibrate in unison or independently to provide one or more of various types of tactile sensations to the user. Alternatively, only a single band element or a subset of band elements may include a vibrotactile.

[0246] Haptic devices 1710, 1720, 1804, and 1930 can include any suitable number and / or type of tactile transducers, sensors, and / or feedback mechanisms. For example, haptic devices 1710, 1720, 1804, and 1930 can include one or more mechanical transducers, piezoelectric transducers, and / or fluid transducers. Haptic devices 1710, 1720, 1804, and 1930 can also include various combinations of transducers of different types and forms that work together or independently to enhance the user's artificial reality experience. In one example, each band element 1932 of haptic device 1930 can include a vibrotactile (e.g., a vibrotactile actuator) that is configured to vibrate in unison or independently to provide one or more of various types of tactile sensations to the user.

[0247] The above embodiments may be implemented in any of a variety of ways. For example, the embodiments may be implemented using hardware, software, or a combination thereof. When implemented using software, the code comprising the software may be executed on any suitable processor or set of processors, whether arranged in a single computer or distributed across multiple computers. It should be understood that any component or set of components that performs the above functions may be generally considered to be one or more controllers that control the above functions. The one or more controllers may be implemented in a variety of ways, such as by dedicated hardware or by using one or more processors programmed using microcode or software to perform the above functions.

[0248] In this regard, it should be understood that one implementation of an embodiment of the present invention includes at least one non-transitory computer-readable storage medium (e.g., computer memory, portable storage, optical disk, etc.) encoded with a computer program (i.e., a plurality of instructions) that, when executed on a processor, performs the functions discussed above of the embodiments of the technology described herein. The computer-readable storage medium may be transportable such that the program stored thereon can be loaded onto any computer resource to implement the various aspects of the present invention discussed herein. Furthermore, it should be understood that reference to a computer program that, when executed, performs the functions discussed above is not limited to an application program running on a host computer. Rather, the term computer program is used herein in a general sense to refer to any type of computer code (e.g., software or microcode) that can be used to program a processor to implement the aspects of the present invention discussed above.

[0249] Various aspects of the technology presented herein may be used alone, in combination, or in various arrangements not specifically discussed in the above embodiments and, therefore, are not limited in their application to the details and arrangements of components set forth in the foregoing description and / or drawings.

[0250] Moreover, some of the above-described embodiments can be implemented as one or more methods, of which some examples have been provided. The actions performed as a part of one or more methods can be sorted in any suitable manner. Therefore, embodiments can be constructed in which the execution order of the actions is different from that shown or described herein, even if shown as sequential actions in the illustrative embodiments, some actions can also be included to be performed simultaneously. The phrases and terms used herein are for descriptive purposes and are not to be considered as restrictive. The use of "including," "comprising," "having," "comprising," "involving," and their variants is intended to include the projects and additional projects listed thereafter.

[0251] Having described several embodiments of the present invention in detail, various modifications and improvements will readily occur to those skilled in the art. Such modifications and improvements are intended to be within the spirit and scope of the present invention. Therefore, the foregoing description is intended to be illustrative only and is not intended to be limiting. The present invention is defined solely by the appended claims and their equivalents.

[0252] The aforementioned features may be used in any of the embodiments discussed herein, alone or together in any combination.

[0253] Furthermore, although advantages of the present invention may be indicated, it should be understood that not every embodiment of the present invention will include every described advantage. Some embodiments may not implement any of the features described herein as advantageous. Therefore, the foregoing description and accompanying drawings are for illustrative purposes only.

[0254] Variations on the disclosed embodiments are possible. For example, various aspects of the present technology may be used alone, in combination, or in various arrangements not specifically discussed in the foregoing embodiments, and therefore are not limited in their application to the details and arrangements of components set forth in the foregoing description or illustrated in the accompanying drawings. Aspects described in one embodiment may be combined in any manner with aspects described in other embodiments.

[0255] The use of ordinal terms (such as "first," "second," "third," etc.) to modify elements in the description and / or claims does not in itself imply any priority, precedence, or order of one element over another, or the temporal order in which the actions of a method are performed, but is merely used as a label to distinguish one element or action with a certain name from another element or action with the same name (but using ordinal terms) to distinguish these elements or actions.

[0256] The indefinite articles "a" and "an" as used in this specification and claims should be understood to mean "at least one" unless explicitly indicated to the contrary.

[0257] Any use of the phrase "at least one" when referring to a list of one or more elements should be understood to mean at least one element selected from any one or more of the elements in the list of elements, but does not necessarily include at least one of each element specifically listed in the list of elements and does not exclude any combination of elements in the list of elements. This definition also allows that elements may optionally be present other than the elements specifically identified in the list of elements to which the phrase "at least one" refers, whether related or unrelated to those specifically identified elements.

[0258] Any use of the phrase "equal" or "same" to refer to two values ​​(such as distances, widths, etc.) means that the two values ​​are the same within manufacturing tolerances. Thus, two values ​​being equal or the same may mean that the two values ​​differ from each other by ±5%.

[0259] The phrase "and / or," as used in this specification and claims, should be understood to mean "either or both" of the elements so connected, i.e., elements that are connected in some cases and elements that are not connected in other cases. Multiple elements listed with "and / or" should be understood in the same manner, i.e., "one or more" of the elements so connected. In addition to the elements specifically identified by the "and / or" clause, other elements may optionally be present, whether related or unrelated to those specifically identified elements. Thus, as a non-limiting example, reference to "A and / or B," when used in conjunction with open language such as "comprising," may, in one embodiment, refer to only A (optionally including elements other than B); in another embodiment, to only B (optionally including elements other than A); in yet another embodiment, to both A and B (optionally including other elements); and so on.

[0260] As used in this specification and claims, "or" should be understood to have the same meaning as "and / or" defined above. For example, when separating items in a list, "or" or "and / or" will be interpreted as including, that is, including at least one of a plurality of elements or a list of elements, and including more than one, as well as optional additional unlisted items. Only terms that clearly indicate the contrary, such as "only one of..." or "exactly one of...", or when used in the claims, "consisting of..." will refer to including exactly one element of a plurality of elements or a list of elements. Generally speaking, when preceded by an exclusive term, such as "any one of," "one of...", "only one of..." or "exactly one of...", the term "or" as used herein will only be interpreted to indicate an exclusive choice (i.e., "one or the other, but not both"). "Substantially consisting of..." when used in the claims should have its ordinary meaning as used in the field of patent law.

[0261] Furthermore, the phrases and terms used herein are for the purpose of description and should not be regarded as limiting. Use of terms such as "including," "comprising," "consisting of," "having," "including," and "involving," and variations thereof herein, is meant to encompass the items listed thereafter and equivalents thereof, as well as additional items.

[0262] The terms "approximately" and "about," if used herein, may be understood to mean within ±20% of a target value in some embodiments, within ±10% of a target value in some embodiments, within ±5% of a target value in some embodiments, and within ±2% of a target value in some embodiments. The terms "approximately" and "about" may be equal to a target value.

[0263] The term "substantially," if used herein, may be understood to mean within 95% of a target value in some embodiments, within 98% of a target value in some embodiments, within 99% of a target value in some embodiments, and within 99.5% of a target value in some embodiments. In some embodiments, the term "substantially" may be equivalent to 100% of a target value.

Claims

1. A computerized system for remotely controlling a device, the system comprising: a plurality of neuromuscular sensors configured to sense neuromuscular signals from a user, the plurality of neuromuscular sensors being arranged on at least one wearable device configured to be worn by the user to obtain the neuromuscular signals; at least one camera capable of capturing information about an environment; as well as at least one computer processor programmed to: generating a map of the environment based on information about the environment captured by the at least one camera and based on first neuromuscular signals sensed by the plurality of neuromuscular sensors, the map including map information, the map information including information for controlling at least one controllable object in the environment, and electronically marking the at least one controllable object in the map based on neuromuscular activity discerned by the at least one computer processor from the first neuromuscular signals sensed by the plurality of neuromuscular sensors; After generating the map of the environment, accessing the map information based on information about the environment captured by at least one camera; as well as After generating the map of the environment, in response to neuromuscular activity discerned from second neuromuscular signals sensed by the plurality of neuromuscular sensors, the at least one controllable object is controlled to change from a first state to a second state based on the information for controlling the at least one controllable object.

2. The system of claim 1, wherein: Map information of the environment is stored in a memory, and After generating the map of the environment, the at least one computer processor retrieves the map information from the memory based on information discerned from information about the environment captured by the at least one camera.

3. The system according to claim 2, wherein: The discerned information is visible in the environment and includes at least one of the following: QR code, Graphic symbols, alphanumeric text string, A 3D object with a specific shape, or A physical relationship between at least two objects.

4. The system according to claim 3, wherein: The identified information includes a reference object of the environment.

5. The system according to claim 1, wherein: The map information includes map data representing a physical relationship between two or more controllable objects in the environment.

6. The system according to claim 5, wherein: The map data is 3D panoramic data of objects in the environment, the objects including the at least one controllable object; and / or wherein the map information includes multiple established center points, and wherein the physical relationship between two or more controllable objects in the environment is determined based on the established center points.

7. The system according to claim 6, wherein: The 3D panoramic data comprises one or more of: a 360° representation of the environment; data relative to a single axis of rotation; and / or a representation of a partial view of the environment.

8. The system according to claim 5, in, The environment is an extended reality (XR) environment including virtual objects and real-world objects, wherein the at least one computer processor is programmed to: determining the position information of the virtual object and the position information of the real-world object based on the map information, and After generating the map of the environment, determining reference objects in the environment based on information about the environment captured by the at least one camera, and The map information includes position information of the at least one controllable object relative to the reference object.

9. The system according to claim 8, wherein: After generating the map of the environment, the neuromuscular activity discerned from the second neuromuscular signals sensed by the plurality of neuromuscular sensors is caused by the user performing the following operations: activating specific motor units relative to the at least controllable object when the user is in the environment; and / or performing at least one gesture relative to the at least controllable object when the user is in the environment.

10. The system according to claim 9, wherein: The at least one gesture includes any one or any combination of the following: The user moves at least one finger relative to the at least one controllable object, the user moving a wrist relative to the at least one controllable object, the user moving an arm relative to the at least one controllable object, The user applies a force but does not move relative to the at least one controllable object, and the user activating a motor unit without movement relative to the at least one controllable object and without force relative to the at least one controllable object; And / or wherein the at least one gesture includes at least one of the following: The user performs a pinching motion with respect to the at least one controllable object using two or more fingers; the user tilting the wrist upward or downward relative to the at least one controllable object; The user moves the arm upward or downward relative to the at least one controllable object; or The user moves the at least one finger upward or downward relative to the at least one controllable object.

11. The system of claim 9, wherein: The at least one controllable object includes a plurality of controllable objects, The at least one gesture comprises a gesture relative to a controllable object among the plurality of controllable objects, and The at least one computer processor is programmed to control each of the plurality of controllable objects to change from a first state to a second state in response to a control signal.

12. A wearable electronic device comprising: a plurality of neuromuscular sensors configured to sense neuromuscular signals from a user, the plurality of neuromuscular sensors being arranged on at least one wearable device configured to be worn by the user to obtain the neuromuscular signals; at least one camera capable of capturing information about an environment; as well as at least one computer processor programmed to: generating a map of the environment based on information about the environment captured by the at least one camera and based on first neuromuscular signals sensed by the plurality of neuromuscular sensors, the map including map information, the map information including information for controlling at least one controllable object in the environment, and electronically marking the at least one controllable object in the map based on neuromuscular activity discerned by the at least one computer processor from the first neuromuscular signals sensed by the plurality of neuromuscular sensors; After generating the map of the environment, accessing the map information based on information about the environment captured by at least one camera; as well as After generating the map of the environment, in response to neuromuscular activity discerned from second neuromuscular signals sensed by the plurality of neuromuscular sensors, the at least one controllable object is controlled to change from a first state to a second state based on the information for controlling the at least one controllable object.

13. The wearable electronic device according to claim 12, wherein: The environment is at least one of the following: Rooms in the home, Rooms in the company, floors of a multi-story building, or Outdoor area.

14. The wearable electronic device according to claim 12, wherein: The at least one controllable object includes at least one of the following: Display devices, electronic game consoles, curtain, lamp, Sound system, lock, or Food or beverage preparation equipment.

15. A computer-implemented method comprising: activating a plurality of neuromuscular sensors configured to sense neuromuscular signals from a user, the plurality of neuromuscular sensors being arranged on at least one wearable device according to any one of claims 12 to 14; activating at least one camera capable of capturing information about the environment; generating a map of the environment based on information about the environment captured by the at least one camera and based on first neuromuscular signals sensed by the plurality of neuromuscular sensors, the map including map information, the map information including information for controlling at least one controllable object in the environment, and electronically marking the at least one controllable object in the map based on neuromuscular activity discerned by the at least one computer processor from the first neuromuscular signals sensed by the plurality of neuromuscular sensors; After generating the map of the environment, accessing the map information based on information about the environment captured by at least one camera; as well as After generating the map of the environment, in response to neuromuscular activity discerned from second neuromuscular signals sensed by the plurality of neuromuscular sensors, the at least one controllable object is controlled to change from a first state to a second state based on the information for controlling the at least one controllable object.

Images