Multi-device interactions for wearable device input

Abstract

According to at least one implementation, a method includes receiving movement data associated with a first computing device and determining a state associated with a second computing device. The method further includes determining that the movement data satisfies at least one criterion associated with an input for the second computing device when in the state. The method also provides causing an action on the second computing device based on the movement data satisfying the at least one criterion.

Description

MULTI-DEVICE INTERACTIONS FOR WEARABLEDEVICE INPUTCROSS-REFERENCE TO RELATED APPLICATION

[0001] This application claims the benefit of U.S. Provisional Application No. 63 / 562,551, filed March 7, 2024, the disclosure of which is incorporated herein by reference in its entirety.BACKGROUND

[0002] An extended reality (XR) device incorporates a spectrum of technologies that blend physical and virtual worlds, including virtual reality (VR), augmented reality (AR), and mixed reality (MR). These devices immerse users in digital environments, either by blocking out the real world (VR), overlaying digital content onto the real world (AR), or blending digital and physical elements seamlessly (MR). XR devices include headsets, glasses, or screens equipped with sensors, cameras, and displays that identify the movement of users and their surroundings to deliver immersive experiences across various applications such as gaming, education, healthcare, and industrial training.SUMMARY

[0003] This disclosure relates to managing input on a wearable device from a second device. In some implementations, a companion device can collect movement data associated with the companion device. In some examples, the companion device can include a smartphone, tablet, watch, or another device. In some implementations, the movement data can be measured from sensors, such as accelerometers, gyroscopes, and magnetometers. These sensors can be used to measure acceleration, orientation, rotation, direction, and / or other movement data. The companion device can be configured to provide the movement data to a second device or wearable device to determine whether the movement data satisfies at least one criterion to trigger an action or an event on the second computing device. In some implementations, the wearable device can execute at least one model (e.g., a machine-learning model) that determines whether the movement corresponds to an input from a user. In some examples, the input can include a tapor taps on the companion device to make a selection action on the wearable device. In some examples, the input can consist of a rotation of the companion device to make a swipe or scroll action on the wearable device.

[0004] In some aspects, the techniques described herein relate to a method including: receiving movement data associated with a first computing device; determining a state associated with a second computing device; determining that the movement data satisfies at least one criterion associated with an input for the second computing device when in the state; and causing an action on the second computing device based on the movement data satisfying the at least one criterion.

[0005] In some aspects, the techniques described herein relate to a computing system including: a computer-readable storage medium; at least one processor operatively coupled to the computer-readable storage medium; and program instructions stored on the computer-readable storage medium that, when executed by the at least one processor, direct the computing system to perform a method, the method including: receiving movement data associated with a first computing device; determining a state associated with a second computing device; determining that the movement data satisfies at least one criterion associated with an input for the second computing device when in the state; and causing an action on the second computing device based on the movement data satisfying the at least one criterion.

[0006] In some aspects, the techniques described herein relate to a computer-readable storage medium having program instructions stored thereon that, when executed by at least one processor, direct the at least one processor to perform a method, the method including: receiving movement data associated with a first computing device; determining a state associated with a second computing device; determining that the movement data satisfies at least one criterion associated with an input for the second computing device when in the state; and causing an action on the second computing device based on the movement data satisfying the at least one criterion.

[0007] The details of one or more implementations are outlined in the accompanying drawings and the description below. Other features will be apparent from the description and drawings and the claims.BRIEF DESCRIPTION OF THE DRAWINGS

[0008] FIG. 1 illustrates a computing environment to manage inputs from a first device to a second device according to an implementation.

[0009] FIG. 2 illustrates a method of managing inputs from a first device to a second device according to an implementation.

[0010] FIG. 3 illustrates an operational scenario of applying a model to movement data and device state to determine an action on a computing device according to an implementation.

[0011] FIG. 4 illustrates an operational scenario of receiving movement data for a first device and updating the state of a second device according to an implementation.

[0012] FIG. 5 illustrates an operational scenario of receiving a tap at a first device and registering the tap as an input for a second device according to an implementation.

[0013] FIG. 6 illustrates an operational scenario of receiving rotational data associated with a first device and registering an input at a second device according to an implementation.

[0014] FIG. 7 illustrates a computing system to manage inputs from a first device to a second device according to an implementation.DETAILED DESCRIPTION

[0015] Computing devices, such as wearable devices and extended reality (XR) devices, provide users an effective tool for gaming, training, education, healthcare, mobile computing, and more. An XR device merges the physical and virtual worlds, encompassing virtual reality (VR), augmented reality (AR), and mixed reality (MR) experiences. These devices can include headsets or glasses equipped with sensors, cameras, and displays that track users’ movements and surroundings, allowing them to interact with digital content in real time. XR devices offer immersive experiences by either completely replacing the real world with a virtual one (VR), overlaying digital information onto the real world (AR), or seamlessly integrating digital and physical elements (MR). Input to XR devices may be provided through gestures, voice commands, controllers, and eye movements. Users interact with the virtual environment by manipulating objects, navigating menus, and triggering actions using these input methods, which are translated by the device’s sensors and algorithms into corresponding digital interactions within the XR space. However, at least one technical problem exists in providing efficient input to a wearable device.

[0016] In at least one technical solution, a system can use a companion device, such as a smartphone, tablet, or watch, to provide user input to a wearable device. In at least one implementation, a wearable device, such as an XR device, can be communicatively coupled with a companion device like a smartphone. A smartphone can include a front-display portion of the device and a back portion, wherein the display and back portions are oriented in opposite directions. The communication between the devices can include a Bluetooth connection, a WIFI connection, or another connection. The companion device can be configured to determine movement data on the device. Movement data can include data from various sensors, such as accelerometers, gyroscopes, and magnetometers. The accelerometer can measure linear acceleration along the X, Y, and Z axes, detecting movement, orientation changes, and subtle vibrations. The gyroscope can track angular velocity, allowing for the detection of rotation and tilt. The magnetometer can sense magnetic fields, helping with compass functions and directional awareness. The captured data from the companion device can be used to determine whether the motion detected at the companion device is directed at causing an action or event in the wearable device.

[0017] In some implementations, an action at the wearable device can trigger a response within a computing device, such as a function execution, state change, or system interaction. In some implementations, an action at the wearable device can trigger a response within a computing device, such as a function execution. In some implementations, an action at the wearable device can trigger a response within a computing device, such as a state change. In some implementations, an action at the wearable device can trigger a response within a computing device, such as a system interaction. An action at the wearable device can be called an event, which triggers a response such as a function execution, state change, or system interaction. In some implementations, an action can be based on an instruction that can be triggered, for example, by a user. In some implementations, an action can be based on an instruction originating at a computing device.

[0018] In some implementations, the companion device determines whether the screen of the companion device is in an off state. If the screen of the companion device is on, the movement data may not correspond to the wearable device. If the screen is in an off state, the user can provide various inputs on the companion device for the wearable device that are detected using the movement data. The inputs can include tapping on the back of the companiondevice, rotating the companion device, or another input directed at causing an action associated with the wearable device. For example, a tap on the back of the companion device can be used to make a selection on the wearable device (e.g., menu selection), change the state of the wearable device (e.g., from inactive to active), or some other operation on the wearable device. As another example, the rotation of the companion device can be used to cause a swiping operation on the wearable device. As at least one technical effect, the companion device can be used to effectively receive input associated with a second device.

[0019] In some implementations, the companion device can be configured to receive (e.g., measure) the movement data using the sensors and provide the movement data to the wearable device for processing. In some implementations, the companion device can provide at least a portion of the processing and calculations associated with the movement data. In some implementations, the system (the wearable device and / or the companion device) can apply a model that determines whether the movement data satisfies a criterion to cause an action. The model can include a machine-learning model configured using a set of movement data and known inputs from the movement data (e.g., movement corresponding to a tap input). In some examples, configuring or training the model involves data collection, preprocessing, feature engineering, model selection, training, evaluation, and deployment. In some examples, raw data is gathered and cleaned to remove inconsistencies (i.e., movement data from a companion device and a desired action from the movement data), which can be followed by normalization and feature extraction to enhance learning efficiency. The dataset can be split into training, validation, and testing subsets to prevent overfitting. A suitable model architecture is selected based on the task (e.g., supervised or unsupervised learning), and optimization techniques such as stochastic gradient descent (SGD) can be employed to minimize a defined loss function. The model can then be validated and deployed to identify intended inputs from the movement data detected by the companion device.

[0020] For example, a wearable device may receive rotational data associated with a gyroscope and / or accelerometer of the companion device. The wearable device can apply the model to the data to determine whether the movement satisfies at least one criterion for a swiping action (e.g., to swipe between windows or digital content). If the data satisfies the at least one criterion, the wearable device can apply the corresponding action. If the data does not satisfy theat least one criterion, then no action will be taken. In some examples, the at least one criterion can include a threshold degree of rotation.

[0021] In some implementations, the system can use multiple machine-learning models to determine whether the movement data measured by the companion device corresponds to an action at the wearable device. In some examples, the selection of the applicable machinelearning model is based on a state of the wearable device. The state of the wearable device can refer to its current or instantaneous status. In some examples, the state of the wearable device can refer to a potential status. In some examples, the state can be based on orientation. In some examples, the state can be based on position. In some examples, the state can be based on motion. In some examples, the state can be based on operational mode (e.g., active or inactive). The state can be determined in some examples from motion data measured at the wearable device or other sensor data associated with the wearable device. In some implementations, the state can comprise an inactive state or an active state. An inactive state can refer to a low-power or standby mode where the device remains on but suspends rendering, and / or sensor processing to conserve energy until reactivated. An active state can refer to an operational model where the device can render visuals, identify user movements, process inputs from sensors, and run applications to support the user experience. In some implementations, the different states of the device can correspond to a different set of actions. For example, when in an inactive state, the system may determine user input via the movement data that indicates that the wearable device should transition to an active state. Alternatively, when in the active state, the system may determine user input via the movement data that indicates various inputs, including a selection of content displayed by the wearable device, a swipe of content displayed by the wearable device, or some other action associated with the wearable device. In some implementations, the input can enable navigation, object manipulation, and communication. In some implementations, the input from the companion device is combined with a determined gaze for the user to make selections for the user. Gaze can be identified using infrared cameras and sensors that determine the movement of the user's eyes. These sensors detect pupil position and direction, allowing the system to determine where the user is looking in real-time. As at least one technical effect, the system can identify the user’s gaze and make selections using the motion data from the companion device.

[0022] FIG. 1 illustrates a computing environment 100 to manage inputs from a first device to a second device according to an implementation. Computing environment 100 includes user 110, wearable device 130, companion device 150, communications 141, and movement data 142. Wearable device 130 includes display 131, sensors 132, camera 133, and input application 126A. Companion device 150 includes display 151, sensors 152, camera 153, and input application 126B. Input application 126A-126B represents a distributed application in some examples, where companion device 150 identifies movement data 142 and provides the movement data 142 to wearable device 130. Companion device 150 can comprise a phone, tablet, smartwatch, or another device. Although demonstrated using a wearable device, similar operations can be performed for other computing systems. For example, a companion device can provide input to a tablet, laptop, desktop computer, or some other computing device based on the companion device’s movement data and the computing device’s state.

[0023] In computing environment 100, device 130 includes display 131, which is a screen or projection surface that presents visual content to user 110, merging virtual elements with the real world. Display 131 can include optical see-through displays (e.g., AR headsets) or video pass-through (e.g., MR / VR devices). Display 131 can use projectors and / or waveguides to display content for user 110. Device 130 further includes sensors 132, such as accelerometers, gyroscopes, magnetometers, depth, infrared, and proximity sensors. The sensors can be used to monitor the physical movement of the user, identify depth information for other objects, identify eye movement for the user, or provide some other operation. Device 130 also includes camera 133, which can capture the real or physical environment to overlay virtual objects (e.g., application interfaces) and identify the movements of user 110 and surroundings to enable accurate interaction within the augmented or virtual space. In some examples, camera 133 can be positioned as an outward view to capture the physical world associated with the user’s gaze. Display 131 can receive updates from input application 126A to implement actions and display content for user 110. Sensors 132 and camera 133 provide data to input application 126A that can be used to identify objects in the physical environment, the location of the objects in the physical environment, or some other information about the user and / or environment. Input application 126A can further receive movement data from companion device 150.

[0024] Companion device 150 includes display 151, sensors 152, camera 153, and input application 126B. Display 151 can be used to show visual content, such as applications, text,images, and videos, using various display technologies, such as liquid crystal displays (LCDs), Organic Light Emitting Diode (OLED), or other display technologies. Sensors 152 can include accelerometers, gyroscopes, magnetometers, proximity sensors, ambient light sensors, fingerprint scanners, GPS, barometers, LiDAR or infrared for depth and facial recognition, or other sensors. Camera 153 can be a built-in front or rear camera for capturing photos and videos or enabling features like facial recognition and augmented reality.

[0025] In some implementations, companion device 150 uses sensors 152 to identify motion or movement data associated with companion device 150. The movement data 142 can be communicated via communications 141 to wearable device 130. In some implementations, companion device 150 and input application 126B determines a status associated with display 151. In some examples, when the display is active (i.e., turned on), the input application 126B may be configured not to provide movement data 142 to wearable device 130. As at least one technical effect, display 151 being active can indicate that the user is providing input to an application on companion device 150 and is not attempting to provide input to companion device 150. In some implementations, wearable device 130 and input application 126A processes movement data 142 to determine whether the movement detected from companion device 150 corresponds to an input. The movement data can indicate a tap on companion device 150, a double tap on companion device 150, a rotation of companion device 150, or some other input from user 110. In some implementations, the tap can correspond to an input or touch on the back of companion device 150 (e.g., opposite display 151). If movement data 142 corresponds to an input or satisfies a criterion associated with an input (e.g., a tap), input application 126A can apply the input to cause an action. Inputs like swipe and tap can be used for navigation, selecting items, scrolling, zooming, switching apps, and triggering actions like opening menus or dismissing notifications. The action caused by the input can be called an event, which triggers a response such as a function execution, state change, or system interaction.

[0026] In some implementations, input application 126A (and input application 126B) can include a model, such as a machine learning model. The model can determine whether movement data 142 satisfies the criteria for action by analyzing input features like acceleration, velocity, and gesture patterns. It can process this data using a trained or configured algorithm, including classification or threshold-based detection, to recognize predefined movement patterns. The model then outputs a decision, such as triggering an action if the movement matches learnedcriteria, using thresholds, statistical analysis, or deep learning-based sequence recognition. For example, the model can be configured to process movement data 142 to determine when the user taps the back of the phone. The model can be configured using a training set that associates movement characteristics with a user’s intent to provide a tap input.

[0027] In some examples, input application 126A can be configured with different models based on the current state of wearable device 130. For example, the first model can be used when the model is in an inactive state, and the second model can be used when the model is in the active state. The inactive state can be used to determine when inputs are provided that are used to transition the state from the inactive state to the active state (e.g., tapping the phone to wake the device). In some examples, the models can consider additional state information, such as display state, to determine whether movement data 142 corresponds to an input.

[0028] In some implementations, at least a portion of movement data 142 can be processed at companion device 150 and the result provided to wearable device 130. For example, companion device 150 can obtain movement data 142 and determine whether movement data 142 satisfies the at least one criterion associated with the state of wearable device 130. If the at least one criterion is satisfied, companion device 150 can communicate a notification to wearable device 130 indicating the result.

[0029] Although demonstrated as exchanging movement data, companion device 150 and wearable device 130 can share additional information. In some examples, the additional information can include a display state associated with companion device 150. The display state can permit wearable device 130 to expand the view of the content from the companion device to the display on the wearable device. In some implementations, companion device 150 can capture one or more images of the user and provide the one or more images to wearable device 130 to provide facial expression information (e.g., for an avatar associated with the user). In some implementations, the companion device 150 or the wearable device 130 can be associated with multiple devices. For example, wearable device 130 can receive input from numerous companion devices. The user can provide voice input to indicate which of the companion devices should be the active device. The user can alternatively place the companion device in a pairing location (e.g., in front of the wearable device camera) to indicate the desired companion device for input.

[0030] FIG. 2 illustrates method 200 of managing inputs from a first device to a second device according to an implementation. The steps of method 200 are referenced parenthetically in the paragraphs that follow with reference to computing environment 100 of FIG. 1.

[0031] Method 200 includes receiving (201) movement data associated with a first computing device and determining (202) a state associated with a second computing device. In some implementations, the first computing device represents a companion device 150, including a smartphone, tablet, watch, or another companion device. In some implementations, the second computing device can represent a wearable device 130, such as smart glasses, an XR device, or another wearable computing device. Companion device 150 can include movement sensors that identify movement associated with the device. The movement data can be derived from sensors, such as accelerometers, gyroscopes, magnetometers, and other movement sensors. In some examples, the movement data includes linear acceleration data. In some examples, the movement data includes angular velocity. In some examples, the movement data includes magnetic field readings. The movement data can be collected and communicated from companion device 150 to wearable device 130. In addition to receiving the movement data, wearable device 130 can be configured to determine a state associated with wearable device 130. The state can correspond to an inactive state or an active state in some examples. In some examples, the state can correspond to one or more applications executing on wearable device 130. In some implementations, the state can correspond to available actions (e.g., buttons) indicated on wearable device 130, wherein the buttons can indicate the type of inputs available (e.g., tap, swipe, etc.).

[0032] Method 200 further includes determining (203) that the movement data satisfies at least one criterion associated with an input for the second computing device when in the state. Method 200 further includes causing (204) an action on the second computing device based on the movement data satisfying the at least one criterion. In some implementations, different states can correspond to different criteria for an input. For example, when the device is in a first state (e.g., inactive), first criteria can be used to define when movement data corresponds to input on wearable device 130. In contrast, when the device is in a second state (e.g., active state), second criteria can define when movement data correspond to input on wearable device 130.

[0033] In some implementations, wearable device 130 can execute different models based on the current state of wearable device 130. Each of the models can include a machine learning model in some examples. A machine learning model is a mathematical representation ofpatterns learned from data used to make predictions or decisions without being explicitly programmed. In some implementations, the models can be trained or configured from different data sets that are used to link movement data to known user inputs (e.g., movement data associated with a tap). In some implementations, a first model can be associated with an inactive state for wearable device 130, and the inputs can be used to wake or provide other inputs to wearable device 130. In some implementations, a second model can be associated with an active state for wearable device 130, and the inputs can be used to provide various inputs, such as tap, swipe, and the like. Each model can be trained on a set of user intent, where movements associated with waking the device (e.g., in an inactive state) can be different from the movements for the device when in the active state. In some implementations, the movement data can be associated with varying input types from the user, including a tap, double tap, swipe, or some other input.

[0034] In some implementations, in determining whether the movement satisfies at least one criterion and causing the action, wearable device 130 can use additional information to determine the user’s intent. In some implementations, the additional information can include the user’s gaze. For example, the model can identify the gaze and determine whether the gaze corresponds to a potential input (e.g., a button displayed on the device). In some implementations, the device can determine a gesture associated with the input using a camera or other sensors to identify the action from the movement data and the gesture. As at least one technical effect, the movement data from companion device 150 can be coupled with the gaze and / or gestures detected from wearable device 130 to determine an input at wearable device 130.

[0035] FIG. 3 illustrates an operational scenario 300 of applying a model to movement data and device state to determine an action on a computing device according to an implementation. Operational scenario 300 includes device 302, and device 303. Device 302 includes sensor data 310. Device 303 includes device state 311, action 312, display 313, model 320, and selection 330.

[0036] In operational scenario 300, device 302 receives sensor data 310 associated with movement at device 302. Device 302 can represent a companion device, such as a smartphone, tablet, smartwatch, or another companion device. Sensor data 310 can include measurements from accelerometers, gyroscopes, and sometimes magnetometers. These sensors provide information on acceleration, angular velocity, and / or orientation, which can be used to identifymotion, position, and rotation in 3D space. Sensor data 310 can be provided to device 303 using a wired or wireless connection and processed using model 320. Device 303 can represent a wearable device, such as smart glasses, an XR device, or some other wearable device. In some implementations, the data is provided when the display for device 302 is inactive, such as in an off or standby state. If the display of device 302 is in an active state, then model 320 may prevent the communication of sensor data 310 to device 303 and model 320. Model 320 can process sensor data 310 with device state 311 to determine an action 312 corresponding to an input provided by the user of device 302 and device 303. In some implementations, model 320 represents a machine-learning model. A machine learning model is a mathematical system that learns patterns from data to make predictions or decisions without being explicitly configured. It can be configured to improve performance over time by adjusting its internal parameters based on new data. In the example of operational scenario 300, the machine-learning model can be configured to use a training set of known inputs and movement data associated with the inputs. The inputs can include a tap, double tap, rotate, or some other inputs at device 302 that can be translated to actions at device 303.

[0037] In some implementations, model 320 can be configured to use device state 311 to determine action 312. In some implementations, device state 311 can indicate whether device 303 is active or inactive. In some implementations, device state 311 can indicate one or more applications that are executing on device 303. In some implementations, device state 311 can indicate a screen state, wherein the screen state can indicate applications that are visible to the user via a display on device 303, visual input elements that are available to the user on device 303, or some other screen state information for device 303. Model 320 can process sensor data 310 and device state 311 to provide action 312, which corresponds to selection 330 on display 313. In some examples, model 320 and device 303 can use supplemental sensor data collected from device 303 to determine whether sensor data 310 should correspond to an action. The supplemental sensor data can include gaze location associated with the user, gestures provided by the user, voice input provided by the user, or other sensor data measured by device 303. Using the example of display 313, the user’s gaze can be used to determine the location of selection 330 based on the tap detected from sensor data 310.

[0038] In some implementations, model 320 can represent a first model from a set of potential models provided by device 303. In some examples, a model can be selected form theset based on device state 31 1 and the sensor data 310 can be processed using the selected model. For example, a first model can be used to identify actions from user input (i.e., movement data) when device 303 is in an inactive state, and a second model can be used to identify actions from user input (i.e., movement data) when device 303 is in an active state.

[0039] FIG. 4 illustrates an operational scenario 400 of receiving movement data for a first device and updating the state of a second device according to an implementation. Operational scenario 400 includes device 402, device 403, user perspective 430, and user perspective 431. Device 402 further includes sensor data 410. Device 403 further includes device state 411, model 420, and device state 412.

[0040] In operational scenario 400, device 402 receives sensor data 410 associated with movement at device 402. Device 402 can represent a companion device, such as a smartphone, tablet, smartwatch, or another companion device. Sensor data 410 can include measurements from accelerometers, gyroscopes, and sometimes magnetometers. These sensors provide information on acceleration, angular velocity, and orientation, which can be used to identify motion, position, and rotation in 3D space. Sensor data 410 can be provided to device 403 using a wired or wireless connection and processed using model 420. Device 403 can represent a wearable device, such as smart glasses, an XR device, or some other wearable device. In some implementations, the data is provided when the display for device 402 is inactive, such as in an off or standby state. If the display of device 402 is in an active state, then model 420 may prevent the communication of sensor data 410 to device 403 and model 420.

[0041] Device 403 can indicate a device state 411 for model 420, wherein device state 411 indicates that device 403 is inactive. An inactive state can refer to a low-power or standby mode where the device remains on but suspends rendering, identification of user movements, and / or sensor processing to conserve energy until reactivated. For example, the inactive state for device 403 may not display content as depicted in user perspective 430. Based on the current device state 411 and sensor data 410, the device can update the state to device state 412 (i.e., active state). Here, the update can change user perspective 430 to user perspective 431, which includes content or playback controls for the user. In some examples, the display may not be updated for the user. In some implementations, model 420 can comprise a machine-learning model configured by associating user intent with movement data (e.g., examples of turning a wearable device active using physical input at the companion device). In some implementations,model 420 is selected from an available set of models based on device 403 being in device state 411. Once transitioned to an active state, device 403 can be configured to use a second model that processes sensor data 410 to implement various actions, such as a selection when a tap is detected at device 402, a scroll when a rotation is detected at device 402, or some other action based on the detected movement data.

[0042] FIG. 5 illustrates an operational scenario 500 of receiving a tap at a first device and registering the tap as an input for a second device according to an implementation. Operational scenario 500 includes device 502, device 503, user perspective 530, and user perspective 531. Device 502 further includes sensor data 510. Device 503 further includes device state 511, model 520, and action 512.

[0043] In operational scenario 500, device 502 receives sensor data 510 associated with movement at device 502. Device 502 can represent a companion device, such as a smartphone, tablet, smartwatch, or another companion device. Sensor data 510 can include measurements from accelerometers, gyroscopes, and sometimes magnetometers. These sensors provide information on acceleration, angular velocity, and orientation, which can be used to identify motion, position, and rotation in 3D space. Sensor data 510 can be provided to device 503 using a wired or wireless connection and processed using model 520. Device 503 can represent a wearable device, such as smart glasses, an XR device, or some other wearable device. In some implementations, the data is provided when the display for device 502 is inactive, such as in an off or standby state. If the display of device 502 is in an active state, then model 520 may prevent the communication of sensor data 510 to device 503 and model 520.

[0044] For operational scenario 500, model 520 is used when device 503 is in device state 511. Device state 511 is associated with user perspective 530 that includes content and a selector 540. In some implementations, the selector can be located on a display based on the user’s gaze. In some implementations, the selector can be located on the display based on a user’s gesture. In some implementations, the selector can be automatically placed over a selectable region of the display. Model 520 processes sensor data 510 and determines that sensor data 510 satisfies at least one criterion in association with action 512. In some implementations, sensor data 510 can indicate the user provides a tap input at device 502. The tap can be detected using a machine-learning model (e.g., model 520) that receives the sensor data 510 and indicatesan input based on sensor data 510. From the input (e.g., tap), action 512 is implemented which implements selection 541 as demonstrated in user perspective 531.

[0045] FIG. 6 illustrates an operational scenario 600 of receiving rotational data associated with a first device and registering an input at a second device according to an implementation. Operational scenario 600 includes device 602, device 603, user perspective 630, and user perspective 631. Device 602 further includes sensor data 610 that registers rotational input 640 (e.g., user rotating the device). Device 603 further includes device state 611, model 620, and action 612.

[0046] Operational scenario 600 provides operations like those discussed above for operational scenario 500. These operations include receiving sensor data 610 from device 602 at device 603 and processing sensor data 610 using model 620. Here, model 620, based on device state 611, determines that the movement of device 602 satisfies at least one criterion and causes action 612, which is a scroll of content demonstrated in the difference from user perspective 630 to user perspective 631. In some implementations, a user of device 602 can rotate the device to implement a scroll associated with the display on device 603. For example, the user can rotate device 602 to scroll through images or a website. In some implementations, model 620 can determine when the orientation rotates a threshold number of degrees. In some implementations, model 620 can include other factors, such as an initial orientation (e.g., facing toward the ground), speed or velocity of the rotation, or some other factors in determining that the movement data satisfies the criteria.

[0047] Although demonstrated in the previous examples, using tap, double tap, and rotation to trigger an action on a wearable device, it should be understood that the movement data can identify other inputs from the user, such as flicks that can be detected via gyroscope and accelerometer data. The movement data can be processed using model to determine whether the movement data corresponds to an input provided from the user.

[0048] FIG. 7 illustrates a computing system 700 to manage cross-device inputs according to an implementation. Computing system 700 represents any apparatus, computing system, or systems with which the various operational architectures, processes, scenarios, and sequences are disclosed herein for managing cross-device input. Computing system 700 can be a wearable device and / or a companion device in some examples. Computing system 700 includes storage system 745, processing system 750, communication interface 760, and input / output (I / O)device(s) 770. Processing system 750 is operatively linked to communication interface 760, I / O device(s) 770, and storage system 745. In some implementations, communication interface 760 and / or I / O device(s) 770 may be communicatively linked to storage system 745. Computing system 700 may further include other components such as a battery and enclosure that are not shown for clarity.

[0049] Communication interface 760 comprises components that communicate over communication links, such as network cards, ports, radio frequency, processing circuitry (and corresponding software), or some other communication devices. Communication interface 760 may be configured to communicate over metallic, wireless, or optical links. Communication interface 760 may be configured to use Time Division Multiplex (TDM), Internet Protocol (IP), Ethernet, optical networking, wireless protocols, communication signaling, or some other communication format - including combinations thereof. Communication interface 760 may be configured to communicate with external devices, such as servers, user devices, or other computing devices.

[0050] I / O device(s) 770 may include peripherals of a computer that facilitate the interaction between the user and computing system 700. Examples of I / O device(s) 770 may include keyboards, mice, trackpads, monitors, displays, printers, cameras, microphones, external storage devices, sensors, and the like. In some implementations, VO device(s) 770 can include one or more sensors that can receive movement data as part of a companion device (e.g., smartphone or watch).

[0051] Processing system 750 comprises microprocessor circuitry (e.g., at least one processor) and other circuitry that retrieves and executes operating software (i.e., program instructions) from storage system 745. Storage system 745 may include volatile and nonvolatile, removable, and non-removable media implemented in any method or technology for storage of information, such as computer-readable instructions, data structures, program modules, or other data. Storage system 745 may be implemented as a single storage device but may also be implemented across multiple storage devices or sub-systems. Storage system 745 may comprise additional elements, such as a controller to read operating software from the storage systems. Examples of storage media (also referred to as computer-readable storage media or a computer- readable storage medium) include random access memory, read-only memory, magnetic disks, optical disks, and flash memory, as well as any combination or variation thereof, or any othertype of storage media. In some implementations, the storage media may be non-transitory. In some instances, at least a portion of the storage media may be transitory. In no case is the storage media a propagated signal.

[0052] Processing system 750 is typically mounted on a circuit board that may also hold the storage system. The operating software of storage system 745 comprises computer programs, firmware, or some other form of machine-readable program instructions. The operating software of storage system 745 comprises input application 724 that is stored on the storage media. The operating software on storage system 745 may further include an operating system, utilities, drivers, network interfaces, applications, or some other type of software. When read and executed by processing system 750 the operating software on storage system 745 directs computing system 700 to operate as described herein. In at least one implementation, the operating software can provide method 200 described in FIG. 2.

[0053] In at least one implementation, input application 724 directs processing system 750 to receive movement data associated with a first computing device and determine a state associated with a second computing device. Input application 724 can further direct processing system 750 to determine that the movement data satisfies at least one criterion associated with an input for the second computing device when in the state and cause an action on the second computing device based on the movement data satisfying the at least one criterion. In some implementations, the processing of the movement data and the state is provided on the second computing device, such as a wearable device. The wearable device can receive the movement data from the companion device. In some implementations, the processing of the movement data and the state can be shared between the devices. In some implementations, input application 724 can include one or more machine-learning models that process the movement data to determine the action. The machine-learning model can be selected to process the movement data based on the state of the second computing device in some examples.

[0054] Example claim clauses are provided below. Although these are examples, these clauses should not be considered exhaustive.

[0055] Clause 1. A method comprising: receiving movement data associated with a first computing device; determining a state associated with a second computing device; determining that the movement data satisfies at least one criterion associated with an input for the secondcomputing device when in the state; and causing an action on the second computing device based on the movement data satisfying the at least one criterion.

[0056] Clause 2. The method of clause 1, further comprising: determining a model from a plurality of models associated with the state; wherein determining that the movement data satisfies the at least one criterion comprises applying the model to the movement data to determine that the movement data satisfies the at least one criterion.

[0057] Clause 3. The method of clause 1, wherein the movement data comprises rotational data for the first computing device, and wherein determining that the movement data satisfies at least one criterion comprises: determining that the rotational data satisfies a threshold degree of rotation.

[0058] Clause 4. The method of clause 1, wherein the state comprises an inactive state, and wherein causing the action on the second computing device comprises causing a change from the inactive state to an active state.

[0059] Clause 5. The method of clause 1, wherein the state comprises an active state, wherein the movement data indicates at least one tap on the second computing device, and wherein the action comprises a selection of content displayed on the second computing device.

[0060] Clause 6. The method of clause 1, wherein determining that the movement data satisfies the at least one criterion comprises: applying a model to the movement data to determine that the movement data satisfies the at least one criterion, the model configured based on additional movement data and at least one action associated with the additional movement data.

[0061] Clause 7. The method of clause 1, wherein applying a model to the movement data to determine that the movement data satisfies the at least one criterion, the model configured based on additional movement data and at least one action associated with the additional movement data.

[0062] Clause 8. The method of clause 1, wherein the state comprises a screen state associated with the second computing device.

[0063] Clause 9. A computing system comprising: a computer-readable storage medium; at least one processor operatively coupled to the computer-readable storage medium; and program instructions stored on the computer-readable storage medium that, when executed by the at least one processor, direct the computing system to perform a method, the methodcomprising: receiving movement data associated with a first computing device; determining a state associated with a second computing device; determining that the movement data satisfies at least one criterion associated with an input for the second computing device when in the state; and causing an action on the second computing device based on the movement data satisfying the at least one criterion.

[0064] Clause 10. The computing system of clause 9, wherein the method further comprises: determining a model from a plurality of models associated with the state; wherein determining that the movement data satisfies the at least one criterion comprises applying the model to the movement data to determine that the movement data satisfies the at least one criterion.

[0065] Clause 11. The computing system of clause 9, wherein the movement data comprises rotational data for the first computing device, and wherein determining that the movement data satisfies at least one criterion comprises: determining that the rotational data satisfies a threshold degree of rotation.

[0066] Clause 12. The computing system of clause 9, wherein the state comprises an inactive state, and wherein causing the action on the second computing device comprises causing a change from the inactive state to an active state.

[0067] Clause 13. The computing system of clause 9, wherein the state comprises an active state, wherein the movement data indicates at least one tap on the second computing device, and wherein the action comprises a selection of content displayed on the second computing device.

[0068] Clause 14. The computing system of clause 9, wherein determining that the movement data satisfies the at least one criterion comprises: applying a model to the movement data to determine that the movement data satisfies the at least one criterion, the model configured based on additional movement data and at least one action associated with the additional movement data.

[0069] Clause 15. The computing system of clause 9, wherein applying a model to the movement data to determine that the movement data satisfies the at least one criterion, the model configured based on additional movement data and at least one action associated with the additional movement data.

[0070] Clause 16. The computing system of clause 9, wherein the movement data indicates a tap on the first computing device wherein the method further comprises: identifying a gaze associated with a user of the second computing device; wherein causing the action on the second computing device comprises causing a selection on the second computing device based on the tap and the gaze.

[0071] Clause 17. A computer-readable storage medium having program instructions stored thereon that, when executed by at least one processor, direct the at least one processor to perform a method, the method comprising: receiving movement data associated with a first computing device; determining a state associated with a second computing device; determining that the movement data satisfies at least one criterion associated with an input for the second computing device when in the state; and causing an action on the second computing device based on the movement data satisfying the at least one criterion.

[0072] Clause 18. The computer-readable storage medium of clause 17, wherein the method further comprises: determining a model from a plurality of models associated with the state; wherein determining that the movement data satisfies the at least one criterion comprises applying the model to the movement data to determine that the movement data satisfies the at least one criterion.

[0073] Clause 19. The computer-readable storage medium of clause 17, wherein the state comprises an active state, wherein the movement data indicates at least one tap on the second computing device, and wherein the action comprises a selection of content displayed on the second computing device.

[0074] Clause 20. The computer-readable storage medium of clause 17, wherein determining that the movement data satisfies the at least one criterion comprises: applying a model to the movement data to determine that the movement data satisfies the at least one criterion, the model configured based on additional movement data and at least one action associated with the additional movement data.

[0075] In this specification and the appended claims, the singular forms “a,” “an” and “the” do not exclude the plural reference unless the context dictates otherwise. Further, conjunctions such as “and,” “or,” and “and / or” are inclusive unless the context dictates otherwise. For example, “A and / or B” includes A alone, B alone, and A with B. Further, connecting lines or connectors shown in the various figures presented are intended to representexample functional relationships and / or physical or logical couplings between the various elements. Many alternative or additional functional relationships, physical connections, or logical connections may be present in a practical device. Moreover, no item or component is essential to the practice of the implementations disclosed herein unless the element is specifically described as “essential” or “critical.”

[0076] Terms such as, but not limited to, approximately, substantially, generally, etc. are used herein to indicate that a precise value or range thereof is not required and need not be specified. As used herein, the terms discussed above will have ready and instant meaning to one of ordinary skill in the art.

[0077] Moreover, the use of terms such as up, down, top, bottom, side, end, front, back, etc. herein are used concerning a currently considered or illustrated orientation. If they are considered concerning another orientation, such terms must be correspondingly modified.

[0078] Further, in this specification and the appended claims, the singular forms “a,” “an” and “the” do not exclude the plural reference unless the context dictates otherwise. Moreover, conjunctions such as “and,” “or,” and “and / or” are inclusive unless the context dictates otherwise. For example, “A and / or B” includes A alone, B alone, and A with B.

[0079] Although certain example methods, apparatuses, and articles of manufacture have been described herein, the scope of coverage of this patent is not limited thereto. It is to be understood that the terminology employed herein is to describe aspects and is not intended to be limiting. On the contrary, this patent covers all methods, apparatus, and articles of manufacture fairly falling within the scope of the claims of this patent.

Claims

WHAT IS CLAIMED IS:

1. A method comprising: receiving movement data associated with a first computing device; determining a state associated with a second computing device; determining that the movement data satisfies at least one criterion associated with an input for the second computing device in the state; and causing an action on the second computing device based on the movement data satisfying the at least one criterion.

2. The method of claim 1, further comprising: determining a model from a plurality of models based on the state; wherein determining that the movement data satisfies the at least one criterion comprises applying the model to the movement data to determine that the movement data satisfies the at least one criterion.

3. The method of claim 1, wherein the movement data comprises rotational data for the first computing device, and wherein determining that the movement data satisfies at least one criterion comprises: determining that the rotational data satisfies a threshold degree of rotation.

4. The method of claim 1, wherein the state comprises an inactive state, and wherein causing the action on the second computing device comprises causing a change from the inactive state to an active state.

5. The method of claim 1, wherein the state comprises an active state, wherein the movement data indicates at least one tap on the second computing device, and wherein the action comprises a selection of content displayed on the second computing device.

6. The method of claim 1, wherein determining that the movement data satisfies the at least one criterion comprises:applying a model to the movement data to determine that the movement data satisfies the at least one criterion, the model configured based on additional movement data and at least one action associated with the additional movement data.

7. The method of claim 1, wherein applying a model to the movement data to determine that the movement data satisfies the at least one criterion, the model configured based on additional movement data and at least one action associated with the additional movement data.

8. The method of claim 1, wherein the state comprises a screen state associated with the second computing device, the screen state indicating content displayed on a display of the second computing device.

9. A computing system comprising: a computer-readable storage medium; at least one processor operatively coupled to the computer-readable storage medium; and program instructions stored on the computer-readable storage medium that, when executed by the at least one processor, direct the computing system to perform a method, the method comprising: receiving movement data associated with a first computing device; determining a state associated with a second computing device; determining that the movement data satisfies at least one criterion associated with an input for the second computing device in the state; and causing an action on the second computing device based on the movement data satisfying the at least one criterion.

10. The computing system of claim 9, wherein the method further comprises: determining a model from a plurality of models associated with the state; wherein determining that the movement data satisfies the at least one criterion comprises applying the model to the movement data to determine that the movement data satisfies the at least one criterion.11 . The computing system of claim 9, wherein the movement data comprises rotational data for the first computing device, and wherein determining that the movement data satisfies at least one criterion comprises: determining that the rotational data satisfies a threshold degree of rotation.

12. The computing system of claim 9, wherein the state comprises an inactive state, and wherein causing the action on the second computing device comprises causing a change from the inactive state to an active state.

13. The computing system of claim 9, wherein the state comprises an active state, wherein the movement data indicates at least one tap on the second computing device, and wherein the action comprises a selection of content displayed on the second computing device.

14. The computing system of claim 9, wherein determining that the movement data satisfies the at least one criterion comprises: applying a model to the movement data to determine that the movement data satisfies the at least one criterion, the model configured based on additional movement data and at least one action associated with the additional movement data.

15. The computing system of claim 9, wherein applying a model to the movement data to determine that the movement data satisfies the at least one criterion, the model configured based on additional movement data and at least one action associated with the additional movement data.

16. The computing system of claim 9, wherein the movement data indicates a tap on the first computing device wherein the method further comprises: identifying a gaze associated with a user of the second computing device; wherein causing the action on the second computing device comprises causing a selection on the second computing device based on the tap and the gaze.

17. A computer-readable storage medium having program instructions stored thereon that, when executed by at least one processor, direct the at least one processor to perform a method, the method comprising: receiving movement data associated with a first computing device; determining a state associated with a second computing device; determining that the movement data satisfies at least one criterion associated with an input for the second computing device in the state; and causing an action on the second computing device based on the movement data satisfying the at least one criterion.

18. The computer-readable storage medium of claim 17, wherein the method further comprises: determining a model from a plurality of models associated with the state; wherein determining that the movement data satisfies the at least one criterion comprises applying the model to the movement data to determine that the movement data satisfies the at least one criterion.

19. The computer-readable storage medium of claim 17, wherein the state comprises an active state, wherein the movement data indicates at least one tap on the second computing device, and wherein the action comprises a selection of content displayed on the second computing device.

20. The computer-readable storage medium of claim 17, wherein determining that the movement data satisfies the at least one criterion comprises: applying a model to the movement data to determine that the movement data satisfies the at least one criterion, the model configured based on additional movement data and at least one action associated with the additional movement data.

Images