Peripheral Devices for Split Computation Architecture

Abstract

1. A method comprising: communicatively coupling a wearable device to a companion device; and mirroring, by the wearable device, data obtained from a peripheral device of the wearable device on the companion device as peripheral data, where the mirroring includes obtaining, by the wearable device, peripheral data from the peripheral device and communicating, by the wearable device, the peripheral data to the companion device, where the method includes receiving, by the companion device, the peripheral data and processing the peripheral data into processed data; transmitting, by the companion device, the processed data to the wearable device; and receiving, by the wearable device, the processed data and completing a computing process utilizing the processed data.

Description

[Technical field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application claims the benefit of and priority to U.S. Provisional Patent Application No. 63 / 363,592, filed April 26, 2022, entitled "SPLIT-COMPUTE ARCHITECTURE," the disclosure of which is incorporated herein by reference in its entirety.

[0002] This application also relates to the related co-pending applications, PCT Application No. PCT / US2023 / 019563, filed April 24, 2023, PCT Application No. PCT / US2023 / 019832, filed April 25, 2023, "SPLIT-COMPUTE ARCHITECTURE" (Attorney Docket No. 0120-497WO1), filed April 26, 2023, "PERIPHERAL DEVICES IN A SPLIT-COMPUTE ARCHITECTURE" (Attorney Docket No. 0120-498WO1), filed April 26, 2023, "MULTIPLE APPLICATION RUNTIMES IN SPLIT-COMPUTE ARCHITECTURE" (Attorney Docket No. 0120-505WO1), filed April 26, 2023, and "MACHINE The disclosure of "LEARNING PROCESSING OFFLOAD IN A SPLIT-COMPUTE ARCHITECTURE" (Attorney Docket No. 0120-509WO1) is incorporated herein by reference.

[0003] Embodiments relate to a wearable device processing architecture. [Background technology]

[0004] Some devices (e.g., wearable devices) may have advanced display capabilities. For these devices, it may be a challenge to fit fully featured electronics into a small form factor. These issues are exacerbated in applications such as accessibility where the device may be expected to be worn all day.

[0005] Existing commercially available systems cannot support continuous usage scenarios. For example, wearable devices (e.g., smart glasses, smart watches, head-mounted displays, etc.) are intended for intermittent engagement and are built around phone-class systems-on-chips (SoCs). These devices can only offer a few hours of battery life with the display turned on. Furthermore, thermal comfort can be an issue due to the small volume of head-mounted displays. Summary of the Invention

[0006] An exemplary implementation includes a wearable device and a companion device that use a split computing architecture. To conserve resources of the wearable device, the companion device can handle computing tasks that would otherwise be performed by the wearable device. The wearable device can include a peripheral device that generates peripheral device data that is stored in memory of the wearable device. The split computing architecture can be configured to facilitate mirroring of peripheral device data on the companion device and / or the wearable device.

[0007] In a general aspect, a device, system, non-transitory computer-readable medium (storing computer-executable program code that can be executed on a computer system), and / or method can be used to perform a process, the method including communicatively coupling a wearable device to a companion device, and mirroring, by the wearable device, data obtained from a peripheral device of the wearable device on the companion device as peripheral data, where the mirroring includes obtaining, by the wearable device, peripheral data from the peripheral device, and communicating, by the wearable device, the peripheral data to the companion device, the method including receiving, by the companion device, the peripheral data and processing the peripheral data into processed data, transmitting, by the companion device, the processed data to the wearable device, and receiving, by the wearable device, the processed data and utilizing the processed data to complete a computing process.

[0008] In another general aspect, a device, system, non-transitory computer-readable medium (storing computer-executable program code that can be executed on a computer system), and / or method can be used to perform a process, the method including: communicatively coupling a wearable device to a companion device; mirroring, by the companion device, data associated with a peripheral device of the wearable device as peripheral data; receiving, by the companion device, the peripheral data from the wearable device; and generating, by the companion device, a result associated with completion of a computing process by the companion device, wherein the computing process is configured to use the peripheral data, and the method includes communicating, by the companion device to the wearable device, the result associated with completion of the computing process.

[0009] Exemplary embodiments will become more fully understood from the detailed description set forth herein below and the accompanying drawings, in which like elements are represented with like reference numerals and which are shown by way of example only and not limitation of the exemplary embodiments, in which: [Brief description of the drawings]

[0010] [Figure 1] FIG. 1 illustrates a block diagram of a high-level split computing architecture in accordance with an exemplary embodiment. [Diagram 2] FIG. 1 illustrates a block diagram of a high-level split computing architecture with a shared runtime environment in accordance with an illustrative embodiment. [Diagram 3] FIG. 1 illustrates a block diagram of a wearable device split computing architecture according to an exemplary embodiment. [Figure 4] FIG. 1 illustrates a block diagram of a high-level split computing architecture in accordance with an exemplary embodiment. [Diagram 5] FIG. 1 illustrates a block diagram of an activity component of a wearable device application according to an exemplary implementation. [Figure 6] FIG. 1 illustrates a block diagram of a wearable device application in a wearable device runtime environment according to an exemplary embodiment. [Figure 7] FIG. 1 illustrates a block diagram of a system using a split computing architecture in accordance with an exemplary embodiment. [Figure 8] FIG. 2 is a block diagram illustrating a companion device runtime environment according to an exemplary implementation. [Figure 9] FIG. 1 is a block diagram of a method of operating a split computing system according to an exemplary embodiment. [Figure 10] FIG. 1 is a block diagram of a method for operating a split computing architecture system according to an exemplary embodiment. [Figure 11] FIG. 1 is a block diagram of a method for operating a companion device according to an exemplary implementation. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] It should be noted that these figures are intended to illustrate general features of methods and / or structures utilized in certain exemplary embodiments and to supplement the written description provided below. However, these figures are not to scale, may not precisely reflect the precise structural or performance characteristics of any given embodiment, and should not be construed to define or limit the range of values ​​or characteristics encompassed by the exemplary embodiments. For example, the arrangement of modules and / or structural elements may be reduced or exaggerated for clarity. The use of similar or identical reference numbers in various figures is intended to indicate the presence of similar or identical elements or features.

[0012] Wearable devices can only provide a few hours of battery life with the display turned on. For example, computationally expensive operations such as image rendering, distortion correction, and location services for wearable display optics may not always be possible to execute efficiently on low-power embedded systems implemented in wearable devices. Thus, display architectures for thin-client wearable devices (e.g., smart glasses) can have the opportunity to reduce the on-device power and thermal footprint by offloading computationally intensive operations (e.g., graphics operations) to companion devices (e.g., mobile devices, smartphones, servers, etc.).

[0013] Some wearable devices may have implementation constraints. For example, implementation constraints for smart glasses may include (1) that the smart glasses need to amplify important services through wearable computing. This may include assistive technologies such as AR and visual recognition. For example, implementation constraints for smart glasses may include (2) that the smart glasses need to last for a full day of use on a single charge. For example, implementation constraints for smart glasses may include (3) that the smart glasses need to look and feel like real glasses. Wearable devices may include augmented reality (AR) devices and virtual reality (VR) devices. Wearable devices may include smart glasses, head-worn devices, and / or head-mounted devices. Wearable devices, head-worn devices, and / or head-mounted devices may be AR / VR devices. A completely standalone wearable device (e.g., smart glasses) solution with a mobile SoC capable of supporting the desired features may not meet the power and industrial design constraints listed above. An on-device computing solution that meets constraints (1), (2), and (3) may be difficult to achieve with current technology, which is limited in that a standalone solution with mobile system-on-chip (SoC) technology that meets the functionality requirements does not meet the power and industrial design constraints listed above.

[0014] A split computing architecture can be used to solve problems associated with implementing existing wearable device technologies that meet the above-mentioned constraints. A split computing architecture can be an architecture that moves the application runtime environment to a remote computing endpoint, such as a smartphone, server, cloud, desktop computer, hereafter often referred to as a companion device. In some implementations, the display content can be streamed back to the wearable device from the companion device. Continuing with the smart glasses example, since the majority of the computation and rendering does not take place on the smart glasses, a split computing architecture can make it possible to utilize low-power processors and / or low-power microcontroller MCU-based systems. In some implementations, a split computing architecture combined with a wearable device that includes an MCU can make it possible to minimize power usage and meet constraints (1) and (2) and (3). New innovations in codecs and networking can sustain the required networking bandwidth in a low-power manner. In some implementations, the wearable device can communicate with the companion device via a well-defined protocol. This architecture can be platform independent.

[0015] In some implementations, peripheral devices such as an inertial measurement unit (IMU) and camera sensors can be configured to generate peripheral data on the wearable device. The companion device can be configured to use the peripheral data in some computing processes. For example, an application running on the companion device can be configured to use peripheral data generated by the IMU of the wearable device. The application could request the peripheral data from the wearable device. However, in an exemplary implementation, the peripheral data can be stored (e.g., in memory) on the companion device. In an exemplary implementation, the wearable device can be configured to communicate (e.g., stream) the peripheral data whenever new peripheral data is available (e.g., automatically without being requested, etc.). In an exemplary implementation, the peripheral data of the wearable device can be mirrored (e.g., replicated memory locations) on the companion device. In an exemplary implementation, the peripheral data of the companion device can be mirrored (e.g., replicated memory locations) on the wearable device.

[0016] FIG. 1 illustrates a block diagram of a high-level split computing architecture according to an exemplary implementation. As shown in FIG. 1, the split computing architecture can include a wearable device 105 including peripheral data 135 and a companion device 110 including peripheral data 140. In an exemplary implementation, the wearable device 105 can be smart glasses, an augmented reality / virtual reality headset, a head mounted display (HMD), a smart watch, a smart ring, and the like. As an example, FIG. 1 illustrates the wearable device 105 as smart glasses 145. In an exemplary implementation, the companion device 110 can be another wearable device, a mobile device, a smartphone, a tablet, a server, and a device including a processor and an operating system. As an example, FIG. 1 illustrates the companion device 110 as a smartphone 150.

[0017] The wearable device 105 and the companion device 110 may be communicatively coupled. For example, the wearable device 105 and the companion device 110 may be communicatively coupled wired or wirelessly. In other words, the wearable device 105 and the companion device 110 may be endpoints of a bidirectional wired and / or wireless communication link. As an example, the wearable device 105 and the companion device 110 may communicate audio and / or video streams over communication line 115. As an example, the wearable device 105 and the companion device 110 may communicate data (e.g., IMU, camera, inputs, etc.) over communication line 120. As an example, the wearable device 105 and the companion device 110 may communicate control streams over communication line 125. The communication line 115, the communication line 120, and / or the communication line 125 may be collectively referred to as communication line 130. In some embodiments, the communication line 130 may be bidirectional.

[0018] In some implementations, the companion device 110 can include a runtime environment to which the wearable device 105 can connect. In some implementations, the wearable device 105 can stream data, such as IMU and camera images, to the runtime environment. In some implementations, the companion device 110 runtime environment can be configured to perform tracking, perception, and / or application rendering, and / or send output, such as encoded video commands or rendering commands, back to the wearable device 105 via a graphical application programming interface (API). In some implementations, the runtime environment can be code or software applications and / or containers that run on the companion device 110. In some implementations, the runtime environment can be software that runs on the companion device 110 simultaneously with the wearable device 105 and / or while the wearable device 105 and the companion device 110 are communicatively coupled. In some implementations, input is captured from the wearable device 105 and injected (e.g., communicated) to the companion device 110 runtime environment.

[0019] In some implementations, the companion device 110 runtime environment can be configured to perform tracking, perception, and application rendering and send output, such as encoded video or rendering commands, back to the wearable device 105 via a graphical application programming interface (API). In some implementations, input is captured from the wearable device 105 and injected (e.g., communicated) to the companion device 110 runtime environment.

[0020] As an example, in the absence of the companion device 110, the computing process would be executed entirely on the wearable device 105. The computing process may be any computing process associated with the functionality of the wearable device. For example, the computing process may be a computing process configured to display content (e.g., as images or video) on a display of the wearable device 105. The computing process may be any computing process that includes computer instructions (e.g., code) stored in a memory of the wearable device 105 that are executed by a processor of the wearable device 105. The computer instructions (e.g., code) may be tasks executed by the processor of the wearable device. In some implementations, the task(s) may be implemented as a service. A service may be, for example, a machine-to-machine interaction over a network. A service may be implemented as a background process. For example, a service may perform network transactions, play audio, perform I / O, interact with a content provider, etc. from the background.

[0021] By including the companion device 110 (e.g., a runtime environment for the companion device 110), the on-board power usage and thermal footprint of the device can be reduced by offloading one or more tasks of the plurality of tasks to the companion device 110. In an exemplary implementation, the plurality of tasks can be fragmented. Fragmenting the tasks can include methodically and / or randomly allocating the plurality of tasks between the wearable device 105 and the companion device 110. For example, the methodical allocation of the plurality of tasks between the wearable device 105 and the companion device 110 can be based on resource usage. For example, if the amount of resources used to execute a task (or plurality of tasks) on the companion device 110 is greater than executing the task (or plurality of tasks) on the wearable device 105 (note that executing a task includes performing a computational operation), then the task (or plurality of tasks) can be executed on the wearable device 105.

[0022] For example, the task (or tasks) may include and / or use peripheral data (e.g., computer data, IMU data, high-resolution images, etc.) captured by a peripheral device of the wearable device 105. If communicating the peripheral data uses more resources (e.g., battery resources of the wearable device 105) than a task involving processing the data by the wearable device 105, the wearable device 105 could be assigned to complete the task. Otherwise, the companion device 110 would need to be assigned to complete the task. In this exemplary implementation, both the wearable device 105 and the companion device 110 can perform the task (or tasks). In some implementations, two or more companion devices are used to complete a process that includes multiple tasks.

[0023] When processing a task, companion device 110 may require data associated with peripheral devices of wearable device 105. Instead of requesting and / or receiving data from wearable device 105, in an exemplary implementation, companion device 110 may read data from peripheral data 140. In other words, in an exemplary implementation, peripheral data 135 is mirrored on companion device 110 as peripheral data 140, so that an application executing on companion device 110 may read data associated with peripheral devices of wearable device 105 from peripheral data 135. Furthermore, in an exemplary implementation, peripheral data 140 is mirrored on wearable device 105 as peripheral data 135, so that an application executing on wearable device 105 may read data associated with peripheral devices of companion device 110 from peripheral data 140.

[0024] In some implementations, when the wearable device 105 is processing a task, the wearable device 105 may need data associated with a peripheral device of the companion device 110. Instead of requesting and / or receiving data from the companion device 110, in an exemplary implementation, the wearable device 105 may read data from the peripheral data 140. In other words, in an exemplary implementation, the peripheral data 140 is mirrored on the wearable device 105 as the peripheral data 135, so that an application running on the wearable device 105 may read data associated with a peripheral device of the companion device 110 from the peripheral data 140. In some exemplary implementations, the peripheral data (or data obtained from the peripheral devices) may include IMU data, image data, audio data, microphone data, multi-channel microphone data, other camera data (e.g., depth data), ambient light sensor (ALS) data, etc.

[0025] FIG. 2 illustrates a block diagram of a high-level split computing architecture with a shared runtime environment according to an exemplary embodiment. As shown in FIG. 2, the wearable device 105 is communicatively coupled to two or more companion devices 110-1, 110-2, ... 110-n via communication lines 130-1, 130-2, ... 130-n, respectively. In some embodiments, the wearable device 105 could roam among the various companion devices 110-1, 110-2, ... 110-n to select the companion device 110-1, 110-2, ... 110-n that provides the best experience at the time. The wearable device 105 can be simultaneously connected to the runtimes of multiple companion devices 110-1, 110-2, ... 110-n. Thus, the wearable device 105 can be simultaneously connected to multiple runtime environments. Additionally, although not shown, peripheral data 135 may be mirrored simultaneously across multiple companion devices 110-1, 110-2, . . . 110-n.

[0026] For example, a runtime environment associated with companion device 110-1 can be configured to project content on the display of wearable device 105, and a runtime environment associated with companion device 110-2 can be configured to access data sources (e.g., sensors) from wearable device 105, database servers, the Internet, etc. for processing. Additionally or alternatively, companion devices 110-1, 110-2,...110-n can be communicatively coupled (e.g., wired or wirelessly) to share resources and / or data. For example, companion device 110-1 and companion device 110-2 can be communicatively coupled to enable a runtime environment associated with companion device 110-1 to receive data from a runtime environment associated with companion device 110-2 for use in the runtime environment associated with companion device 110-2 generating images (or frames) and projecting content on the display of wearable device 105.

[0027] For example, companion device 110-1 and companion device 110-2 may be communicatively coupled (e.g., via communication link 205) to allow a runtime environment associated with companion device 110-2 to share processing resources with a runtime environment associated with companion device 110-1. For example, a runtime environment associated with companion device 110-1 may be instructed to generate content (e.g., images, frames, maps, etc.) that can be more efficiently generated by a runtime environment associated with companion device 110-2. For example, companion device 110-2 may include a map database and a map (e.g., image) generator. The runtime environment associated with companion device 110-1 may be instructed to generate content that includes a map. In this example, the runtime environment associated with companion device 110-1 may request a map from a runtime environment associated with companion device 110-2.

[0028] Continuing with the above example, once a determination is made that companion device 110-1 needs to perform a task(s), companion device 110-1 (or wearable device 105) may determine that two or more companion devices 110 can perform the task(s). For example, the task(s) may include generating content (e.g., images) to be displayed on wearable device 105. In this example, companion device 110-1 may be configured to generate the content. However, data used to generate the content may be generated by companion device 110-2 and then communicated (e.g., via communication line 205) from companion device 110-2 to companion device 110-1. For example, companion device 110-2 may be a wearable smartwatch configured to sense a user's heart rate. Data representative of the heart rate may be communicated from companion device 110-2 to companion device 110-1. Companion device 110-1 can then generate content using the data representing the heart rate, and the content can be communicated to wearable device 105 for display on a display of wearable device 105.

[0029] FIG. 3 illustrates a block diagram of a wearable device split compute architecture according to an exemplary embodiment. As shown in FIG. 3, the wearable device split compute architecture can include a hardware abstraction layer (HAL) 310 block. The HAL 310 can be a layer of software configured to interface between an operating system (e.g., RTOS 340) and hardware devices at a general or abstract level rather than at a hardware level. In some embodiments, the use of an abstraction layer can make the split compute architecture platform independent. The HAL 310 can be called from an operating system kernel. In some embodiments, the HAL 310 can be a virtual HAL. The virtual HAL can minimize interpretation delays based on architectural similarities between the guest and host platforms. Virtualization techniques help map virtual resources to physical resources and use native hardware for computation within the virtual HAL.

[0030] Thus, by way of example, HAL 310 may include a connectivity 315 block, a codec 320 block, a graphics processing unit (GPU) 325 block, and a display 330 block, each configured to interface with a corresponding hardware device. For example, connectivity 315 may be configured to interface between an operating system (e.g., RTOS 340) and Bluetooth hardware, WIFI hardware, ultra-wideband (UWB) hardware, 5G hardware, etc. Thus, the wearable device split-computation architecture may be configured to utilize any connectivity hardware designed into wearable device 105.

[0031] For example, the codec 320 can be configured to interface between an operating system (e.g., RTOS 340) and an encoder and / or decoder (e.g., hardware-based encoder and / or decoder). The codec 320 standard can be, for example, H.265, H.264, MPEG, VP9, ​​machine-learned, etc. The codec 320 can be an image, video, and / or audio codec. Thus, the wearable device split-computation architecture can be configured to utilize any codec software and / or hardware designed into the wearable device 105. For example, the GPU 325 can be configured to interface between an operating system (e.g., RTOS 340) and GPU hardware (e.g., GPU ASIC). Thus, the wearable device split-computation architecture can be configured to utilize any GPU designed into the wearable device 105 (e.g., to render images on a display system). For example, the display 330 can be configured to interface between an operating system (e.g., RTOS 340) and display hardware (e.g., the wearable device display). Thus, the wearable device split-computation architecture can be configured to utilize any display (e.g., display driver system) designed into the wearable device 105.

[0032] As shown in FIG. 3, the wearable device split computing architecture can include an Operating System (OS) Abstraction Layer (OSAL) 335. In some implementations, the use of an abstraction layer can make the split computing architecture platform independent. The OSAL 335 can be configured to provide an interface to common system functions provided by the OS of the wearable device 105. These OSALs 335 can simplify developing and porting software (e.g., applications) to multiple OS and hardware platforms. In some implementations, the OSAL 335 can operate as (or similar to) an Application Programming Interface (API). In some implementations, the OSAL 335 can be platform dependent.

[0033] OSAL 335 may include a block of real-time operating system (RTOS) 340. RTOS 340 may be configured to handle, for example, multi-threaded applications to meet real-time deadlines. For example, RTOS 340 may be configured to handle multiple tasks, each (or group of tasks) having a maximum completion time. Although an RTOS is shown, any OS may be used. For example, a high-level operating system (HLOS) may be used. In a split computing system, the use of tasks allows these tasks (or groups of tasks) to be distributed among computing devices. For example, content display operations for wearable device 105 may be split between wearable device 105 and companion device 110 (e.g., a runtime environment associated with companion device 110). In other words, wearable device 105 (and / or companion device 110) may be configured (e.g., using a wearable device split computing architecture) to have companion device 110 perform a portion (task or group of tasks) of a process (e.g., content display). Causing the device to perform a task may include transmitting instructions configured to initiate and / or trigger processing of the task by the device and / or another device(s). A computing process may be initiated by a user of the wearable device providing a corresponding user input (e.g., a gesture) to the wearable device. In some implementations, the wearable device may initiate a computing process based on another computing process, based on a spatial position of the wearable device, etc.

[0034] 3, the wearable device split-computing architecture may include a peripheral driver 345 block. The peripheral driver 345 may be configured to interface between the OS and peripheral devices. For example, the wearable device 105 may include multiple peripheral devices including, for example, camera(s), microphone(s), speaker(s), input(s), inertial measurement unit(s) (IMU), etc. Thus, the peripheral driver 345 may be configured to interface between the RTOS 340 and the peripheral device(s) of the wearable device 105.

[0035] As shown in FIG. 3, the wearable device split computing architecture can include a device client 305. The device client 305 can be configured to control communication between the wearable device 105 and the companion device 110. For example, the device client 305 can be configured to create, initialize, and control the communication line 130 and the communication over the communication line 130. The communication line 130 can operate, for example, as a socket (e.g., a network socket, a TCP / IP network socket, etc.). A socket can be one endpoint of a bidirectional communication link between computer code (e.g., applications, programs, software systems, etc.) running on two computing devices. For example, a two-way handshake can be used to indicate that both the client and the host determine which functions each supports. The socket mechanism can be configured to provide inter-process communication (IPC) by establishing named communication contacts between the two endpoints and / or between the two endpoints and an intermediate device (e.g., an access point (AP)). The socket can be configured to provide a bidirectional first-in, first-out (FIFO) communication channel. A socket that connects to the network is created on each end of the communication. As an example, each socket can have an address (or memory location). The address (or memory location) can be, for example, an IP address and a port number. Thus, the device client 305 can be configured to write to and read from a socket associated with the companion device 110 (or a runtime environment associated with the companion device 110). In some implementations, the device client 305 can be platform independent.

[0036] In some implementations, a software development kit (SDK) may be associated with the wearable device 105 and the companion device 110. The SDK may be used when developing applications for the wearable device 105 and / or the companion device 110. The SDK may enable implementation of a split computing architecture. Thus, any wearable device 105 and / or companion device 110 that includes a split computing architecture may use applications developed using the SDK (regardless of the hardware and / or software platform). Thus, an application does not need to be developed and ported to each hardware and / or software platform that may be used as the wearable device 105 and / or the companion device 110. The SDK may be included (or have elements that may be included) in an application when the application is installed on the wearable device 105 and / or the companion device 110.

[0037] FIG. 4 illustrates a block diagram of a high-level split computing architecture according to an exemplary implementation. As shown in FIG. 4, the system can include a wearable device 105 and a companion device 110. The split computing architecture of the system can include a device client 305 block associated with the wearable device 105, and an application 410 block, an SDK 415 block, and a core 420 block associated with the companion device 110. As described above, the device client 305 can be configured to control communications between the wearable device 105 and the companion device 110. The core 420 can be configured to control communications between the wearable device 105 and the companion device 110. Thus, the core 420 can be configured to at least create, initialize, and control the communication line 130 and communications over the communication line 130. The communication line 130 can operate, for example, as a socket (as described above).

[0038] In some implementations, application 410 may include a file format for applications used on an OS that holds application logic (e.g., Android Package Kit (APK)). In some implementations, the wearable device application may link with a stub (e.g., not full) version of the SDK to support compilation and testing services. In some implementations, at runtime, application 410 may load wearable device SDK 415 directly from the wearable device runtime environment. In some implementations, SDK 415 may provide developers with application programming interfaces (APIs) used to build wearable device applications. In some implementations, an API versioning scheme allows for new APIs to be introduced while maintaining backward compatibility. In some implementations, the wearable device runtime environment may be a collection of core services that are responsible for maintaining the wearable device execution environment. In some implementations, the wearable device application may not directly interact with the core services. In some implementations, one or more interactions may pass through the SDK. In some implementations, the device client 305 may be a thin client that runs on the hardware of the wearable device 105.

[0039] As an example, the application 410 can be configured to generate (or help generate) content for display on the wearable device 105. For example, the application can be configured to process a task(s). For example, the application can generate content (e.g., as an image) and communicate the content to the core 420 via the SDK 415. Communicating the content to the core 420 via the SDK 415 may be one of the features that allows the application 410 to be developed for any hardware and / or software platform. For example, the SDK 415 can be configured to communicate with the application 410 as the application 410 is developed. The SDK 415 can also be configured to communicate with the core 420 associated with multiple hardware and / or software platforms. After receiving the content, the core 420 can communicate the content to the wearable device 105 via the device client 305, for example, using a pre-established socket. The application 410 can be configured to generate the content based on data received from the wearable device 105. For example, the peripheral driver 345 can sense data and communicate the data as peripheral data (e.g., IMU data) to the application module 630 via the device client 305 and core 420 using the communication lines 120. In other words, peripheral data can be data that is collected and processed (e.g., compressed, packaged, parsed, filtered, denoised, etc.) by the peripheral device via the peripheral driver 345 and packaged for communication with and use by the wearable device 105 and / or companion device 110.

[0040] As mentioned above, in some implementations, the wearable device 105 may be configured to connect to multiple companion devices 110 at a given time. In some implementations, different companion devices 110 may be configured to provide different services (e.g., using applications 410). In some implementations, the companion device 110 may be configured to operate in the cloud (e.g., connect through the 5G standard) as low-latency, high-bandwidth 5G connections become mainstream.

[0041] 5 illustrates a block diagram of an activity component of a wearable device application according to an exemplary implementation. As shown in FIG. 5, application 410 can include a wearable activity 505 block, a wearable activity services 510 block, and a wearable activity host 515 block, and core 420 can include a core services 520 block.

[0042] In some example implementations, the design of a wearable device application may resemble an activity model. Referring to Figure 3, the RTOS 340 may be configured to handle one task, multiple tasks, each (or group of tasks) having a maximum time to complete. Thus, each activity may be a task that runs in parallel processes and has a time (or amount of time) to complete.

[0043] In an exemplary implementation, the activity can be executed in a service context. In some implementations, executing in a service context allows the application 410 to run concurrently with applications on the companion device 110. In some implementations, the application can continue to run and render when the display of the companion device 110 is turned off.

[0044] In some implementations, one or more of the applications 410 can include a wearable activity service 510. The wearable activity service 510 can be configured to expose a wearable activity 505 so that the wearable activity 505 can be instantiated by the SDK 415. Thus, the wearable activity 505 can be instantiated at a later point in time. From a developer's perspective, the wearable activity service 510 can be boilerplate code that is not directly related to the logic of the application 410. The wearable activity 505 can be code that is related to the logic of the application 410. In some implementations, the wearable activity 505 can be managed by an activity manager and can behave similarly to a standard OS activity counterpart.

[0045] In some implementations, service binding can be used by the wearable device runtime environment to initiate and manage the lifecycle of application 410. Once the activity manager binds the service as part of the launch flow, SDK 415 can instantiate and attach wearable activity host 515 as a class that can, for example, be responsible for general activity state control. For example, during initialization, wearable activity host 515 can request a surface from the window manager. This surface is then used as a backing store for the virtual display used to render the content of application 410.

[0046] FIG. 6 illustrates a block diagram of a wearable device application in a wearable device runtime environment according to an exemplary implementation. As illustrated in FIG. 6, the companion device OS 605 can handle two or more application environments simultaneously. For example, the companion device OS 605 can include an application module 610. The application module 610 can be associated with standard OS application activities. Additionally, the companion device OS 605 can include an application module 630 that operates in association with a wearable runtime environment 625. The wearable runtime environment 625 can be associated with the wearable device 105.

[0047] An activity(ies) 620, 640 can be a single focused task that an application can perform. For example, some applications can include a user interface (UI). Thus, the activity(ies) 620, 640 can be configured to create a window in which to place the UI. The window can be a full screen window, a floating window embedded in another window, a hidden window, etc. Different types of windows can be associated with different activity(ies) 620, 640. The activity(ies) 620, 640 can be configured for any task, and windows are just one example.

[0048] The activity managers 615, 635 can be configured to communicate information about and interact with the activities 620, 640. The activity managers 615, 635 can also be configured to communicate information about and interact with tasks, threads, services, and other processes.

[0049] In an exemplary implementation, the wearable runtime environment 625 may be a virtual runtime environment. A virtual runtime environment may be configured to run in the background of a computing device. Thus, the wearable runtime environment 625 may be a virtual runtime environment associated with the wearable device 105 and configured to run as a background process on the companion device 110. In other words, the wearable runtime environment 625 may run without a user interface shown on the display of the companion device 110. For example, the wearable runtime environment 625 may run in a hidden window on the companion device 110. Thus, if the wearable runtime environment 625 is a virtual runtime environment, a user of the companion device 110 may not be able to visually control or I / O control an application using the wearable runtime environment 625.

[0050] In alternative or additional implementations, the application 410 and / or application module 630 may be a virtual process. A virtual process may be configured to run in the background of a computing device. Thus, the application 410 and / or application module 630 may be a virtual process associated with the wearable device 105 and configured to run as a background process on the companion device 110. In other words, the application 410 and / or application module 630 may run without a user interface shown on the display of the companion device 110. For example, the application 410 and / or application module 630 may run in a hidden window on the companion device 110. Thus, if the process is a virtual runtime process, a user of the companion device 110 may not have visual or I / O control over the application 410 and / or application module 630.

[0051] FIG. 7 illustrates a block diagram of a system using a split computing architecture according to an exemplary implementation. As shown in FIG. 7, the system includes a wearable device 105 and a companion device 110. The wearable device 105 may include a device client 305, a mirror module 705, and a peripheral driver 345. The mirror module 705 may include peripherals 715-1, 715-2, 715-3, ..., and peripherals 715-n. Each of the peripherals 715-1, 715-2, 715-3, ..., and peripherals 715-n may include peripheral data. The companion device 110 may include a core 420 and a wearable runtime environment 625. The wearable runtime environment 625 may include an application module 630 including an application 410, and a mirror module 710. The mirror module 710 can include peripherals 720-1, 720-2, 720-3, ..., and peripherals 720-n. Each of the peripherals 720-1, 720-2, 720-3, ..., and peripherals 720-n can include peripheral data. This example implementation can be used to explain signal flows associated with the companion device 110 mirroring and using peripheral data.

[0052] In this exemplary implementation, application 410 can be configured to generate, for example, content, head poses, rendered images, etc. As an example, generating a head pose (e.g., head pose manipulation) can include using images and / or IMU data captured by wearable device 105. In this example, when a camera (e.g., a peripheral device) captures an image as image data, the image data can be stored, for example, in peripheral device 715-1 as peripheral data. Additionally, when an IMU senses data, the IMU data can be stored, for example, in peripheral device 715-2 as peripheral data.

[0053] In this example implementation, peripherals 715-1, 715-2, 715-3, ..., and peripherals 715-n may be mirrored on companion device 110 as peripherals 720-1, 720-2, 720-3, ..., and peripherals 720-n, respectively. Mirroring, data mirroring, or mirrored data may include copying data from a first location to a second location in real time. Because the data is copied in real time, the data stored in the second location is an exact copy of the data in the first location. The first location and the second location may be memory locations (e.g., addressable memory locations). In an example implementation, the first location may be a memory of wearable device 105 and the second location may be a memory of companion device 110 (e.g., a memory associated with wearable runtime environment 625). Continuing with the above example, peripheral device 720-1 is a mirror (e.g., an exact copy) of peripheral device 715-1, and peripheral device 720-2 is a mirror (e.g., an exact copy) of peripheral device 715-2. Thus, the peripheral data for peripheral device 720-1 is the same peripheral data for peripheral device 715-1, and the peripheral data for peripheral device 720-2 is the same peripheral data for peripheral device 715-2.

[0054] Thus, application 410 can be configured to use peripheral data of peripheral device 720-1 and peripheral data of peripheral device 720-2 to generate, for example, content, head pose, rendered images, etc. The generated content, head pose, rendered images, etc. should be substantially the same if the process is performed by wearable device 105, thus conserving resources of wearable device 105.

[0055] 8 is a block diagram illustrating a companion device runtime environment 625 according to an exemplary implementation. The runtime environment 625 can be included in a memory of the companion device 110. The memory can include both volatile memory (e.g., RAM) and non-volatile memory, such as one or more ROMs, disk drives, solid-state drives, etc. In an exemplary implementation, the runtime environment 625 can include modules (e.g., software code, computer instructions, etc.) configured to generate head pose data. The head pose data can be generated based on image data and / or IMU data.

[0056] In some implementations, one or more of the components of the runtime environment 625 may be associated with a processor configured to process instructions stored in a memory. Examples of such instructions shown in Figure 8 include a mirror module 710, an IMU manager 810, a neural network manager 820, and a visual positioning system manager 830. Additionally, as shown in Figure 8, the runtime environment 625 may be configured to store various data, which will be described with respect to each module that uses such data.

[0057] In this example, the mirror module 710 includes an image peripheral 805-1 and an IMU peripheral 805-2, each of which includes peripheral data that is a mirror of the peripheral data on the wearable device 105. The IMU manager 810 can be configured to acquire the IMU data 850. In some implementations, the IMU manager 810 can be configured to acquire the IMU data 850 by reading the peripheral data from the IMU peripheral 805-2. As shown in FIG. 8, the IMU manager 810 can include an error compensation manager 812 and an integration manager 814.

[0058] The error compensation manager 812 can be configured to store the IMU calibration parameters. The error compensation manager 812 can be further configured to receive the IMU output (IMU data 850) from the IMU manager 810, for example, and compensate the IMU output for errors using the IMU calibration parameter values. The error compensation manager 812 can be further configured to generate the IMU data 850 after performing the error compensation.

[0059] The integration manager 814 can be configured to perform integration operations (e.g., summing time-dependent values) on the IMU data 850. For example, the rotational rate data 851 can be integrated over time to generate an orientation. Additionally, the acceleration data 852 can be integrated twice over time to generate a position. Thus, the integration manager 814 can be configured to generate a 6DoF pose (position and orientation) from the IMU outputs, such as the rotational rate data 851 and the acceleration data 852.

[0060] The IMU data 850 may represent gyro and accelerometer measurements, rotation rate data 851, and acceleration data 852 in a world frame (e.g., as opposed to a local frame, such as the frame of the IMU) compensated for error(s) using IMU calibration parameter values. Additionally, the IMU data 850 includes 6DoF attitude and movement data, position data 854, orientation data 855, and velocity data 856 derived from the gyro and accelerometer measurements. Finally, in some implementations, the IMU data 850 may include IMU temperature data 853, which may indicate further errors in the rotation rate data 851, and acceleration data 852 (for correction by the error compensation manager 812).

[0061] Neural network manager 820 can be configured to take as input rotational rate data 851 and acceleration data 852 and generate neural network data 840 including first position data 842, first orientation data 844, and first velocity data 846. In some implementations, the input rotational rate data 851 and acceleration data 852 can be generated by error compensation manager 812 operating on raw IMU output values, for example, with error compensated by IMU calibration parameter values. As shown in FIG. 8, neural network manager 820 can include a neural network training manager 822.

[0062] Neural network training manager 822 can be configured to receive training data 848 and generate neural network data 840, including data regarding layers and cost functions and values. In some implementations, training data 848 can include movement data generated from measurements of a user wearing wearable device 105 and moving, for example, their head and / or other parts of their body, as well as ground truth 6DoF pose data generated from those measurements. In some implementations, training data 848 can include measured rotational velocities and accelerations from motion combined with measured 6DoF poses and velocities.

[0063] Additionally, in some implementations, the neural network manager 820 can use historical data from the IMU to generate the first position data 842, the first orientation data 844, and the first velocity data 846. For example, the historical data can be used to augment the training data 848 with previous rotational speeds, accelerations, and temperatures for the resulting 6DoF attitude and movement results, thus further refining the neural network. In some implementations, the neural network represented by the neural network manager 820 is a convolutional neural network, and the layers are convolutional layers.

[0064] A visual positioning system (VPS) manager 830 may be configured to receive an image as an input and generate VPS data 860 including second position data 862 and second orientation data 864. In some implementations, the VPS data may include second velocity data 866, such as a 6DoF pose based on the image. The VPS manager 830 may be configured to acquire the image used to generate the VPS data 860. In some implementations, the VPS manager 830 may be configured to acquire the image by reading peripheral data from an image peripheral 805-1. The image or image data stored in the image peripheral 805-1 may be a mirror of an image captured by a world-facing camera on the wearable device 105.

[0065] In some implementations, the level of accuracy of VPS manager 830 in generating VPS data 860 may depend on the environment surrounding the location. For example, the accuracy requirement for an indoor location may be on the order of 1-10 cm, while the accuracy requirement for an outdoor location may be on the order of 1-10 m.

[0066] The head pose of the wearable device 105 can be generated by the companion device 110 based on the peripheral data of the image peripheral 805-1 and / or the IMU peripheral 805-2. For example, the head pose of the wearable device 105 can be generated by the companion device 110 based on the neural network data 840. For example, the head pose of the wearable device 105 can be generated by the companion device 110 based on the VPS data 860. For example, the head pose of the wearable device 105 can be generated by the companion device 110 based on the IMU data 850. For example, the head pose of the wearable device 105 can be generated by the companion device 110 based on the neural network data 840, the VPS data 860, and / or the IMU data 850. The generated head pose should be substantially the same if the process is performed by the wearable device 105, thus saving resources of the wearable device 105. Furthermore, due to the split computing system and mirrored peripheral data of the imaging peripheral 805-1 and / or the IMU peripheral 805-2, the head pose can be generated within a maximum completion time.

[0067] Example 1. FIG. 9 is a block diagram of a method of operating a split computing system including a wearable device and a companion device communicatively coupled to the wearable device, according to an exemplary embodiment. As shown in FIG. 9, in step S905, the wearable device is communicatively coupled to the companion device. The coupling can be triggered by either the wearable device, the companion device, or in response to a user command of the wearable device or of a user of the companion device. The wearable device can be "communicatively coupled" to the companion device if the wearable device is capable of sending one or more commands and / or data, at least in part, to and / or receiving one or more commands and / or data from the companion device, such as, for example, via one or more wired and / or wireless communication links. In some embodiments, a two-way handshake can be used, indicating that both the client and the host determine which functions each supports. In step S910, mirroring data acquired from a peripheral device by the wearable device at the companion device as peripheral data, the mirroring including acquiring the peripheral data from the peripheral device by the wearable device and communicating the peripheral data to the companion device by the wearable device. Here, the term "peripheral device" may be used to refer to an internal device of the wearable device or an external device that directly connects to the wearable device, such as an input / output device of the wearable device, such as an inertial measurement unit (IMU), etc. Here, the term "mirroring" may be used to refer to the real-time act of copying the peripheral data to the companion device as an exact copy. In step S915, receiving the peripheral data by the companion device and processing the peripheral data into processed data.The mirrored peripheral data may relate to multiple tasks, each (or group of tasks) potentially having a maximum completion time, where the multiple tasks or a portion of the multiple tasks may be executed by the companion device when mirroring the peripheral data to the companion device. Results of the completed tasks may then be returned to the wearable device. In step S920, the companion device transmits the processed data to the wearable device. In step S925, the wearable device receives the processed data and utilizes the processed data to complete a computing process, where the computing process may begin before or when the peripheral data is transmitted to the user, e.g., when the wearable device is coupled with the companion device.

[0068] Example 2. The method of example 1, further comprising: mirroring, by the wearable device, data obtained from a peripheral device of the companion device on the wearable device as companion device peripheral data, comprising receiving, by the wearable device, the companion device peripheral data from the companion device.

[0069] Example 3. The method of example 1, wherein the peripheral data may be inertial measurement unit (IMU) data, and the computing process may be a head pose manipulation, and completing the computing process by the wearable device may include using the result based on the head pose manipulation.

[0070] Example 4. The method of example 1, wherein the peripheral data may be image data, and the computing process may be a head pose manipulation, and completing the computing process by the wearable device may include using the result based on the head pose manipulation.

[0071] Example 5. The method of example 1, wherein the peripheral data may be image data, and the computing process may be an eye-tracking operation, and completing the computing process by the wearable device may include using the result based on the eye-tracking operation.

[0072] Example 6. The method of example 1, wherein the wearable device can include a first socket, the companion device can include a second socket communicatively coupled to the first socket, and mirroring the data associated with a peripheral device can include communicating the peripheral data between the first socket and the second socket.

[0073] Example 7. The method of example 1, wherein the wearable device can be smart glasses.

[0074] Example 8. The method of example 1, wherein the companion device can be at least one of another wearable device, a mobile device, a smartphone, a tablet, a server, and a device that includes a processor and an operating system.

[0075] Example 9. FIG. 10 is a block diagram of a method for operating a split computing architecture according to an exemplary embodiment. The system may include a wearable device including a device client, a hardware application layer, an operating system abstraction layer, and at least one peripheral device driver. The system may include a companion device including a runtime environment associated with the wearable device. As shown in FIG. 10, in step S1005, data associated with the peripheral device is mirrored on the companion device as peripheral data. In step S1010, the wearable device detects the peripheral data. In step S1015, the wearable device communicates the peripheral data to the companion device. In step S1020, the companion device receives the peripheral data. In step S1025, the companion device generates a result associated with the completion of a computing process by the companion device, and the computing process is configured to use the peripheral data. In step S1030, a result associated with the completion of the computing process is received by the wearable device, and in step S1035, the computing process is completed by the wearable device based on the result associated with the completion of the computing process.

[0076] Example 10. The method of example 9, wherein the wearable device can be smart glasses.

[0077] Example 11. The method of example 9, wherein the companion device can be at least one of another wearable device, a mobile device, a smartphone, a tablet, a server, and a device that includes a processor and an operating system.

[0078] Example 12. The method of example 9, wherein the runtime environment may be a virtual runtime environment that runs in the background of the companion device, and a mirror of the data associated with the peripheral device may be included in the virtual runtime environment.

[0079] Example 13. The method of example 9, wherein the peripheral data may be inertial measurement unit (IMU) data, and the computing process may be a head pose manipulation, and completing the computing process by the wearable device may include using the result based on the head pose manipulation.

[0080] Example 14. The method of example 9, wherein the peripheral data may be image data, and the computing process may be a head pose manipulation, and completing the computing process by the wearable device may include using the result based on the head pose manipulation.

[0081] Example 15. The method of example 9, wherein the peripheral data may be image data, and the computing process may be an eye-tracking operation, and completing the computing process by the wearable device may include using the results based on the eye-tracking operation.

[0082] Example 16. The method of example 9, wherein the wearable device can include a first socket, the companion device can include a second socket communicatively coupled to the first socket, and mirroring the data associated with a peripheral device can include communicating the peripheral data between the first socket and the second socket.

[0083] Example 17. Figure 11 is a block diagram of a method of operating a companion according to an exemplary embodiment. As shown in Figure 11, in step S1105, a wearable device is communicatively coupled to a companion device. In step S1110, the companion device mirrors data associated with a peripheral device of the wearable device as peripheral data. In step S1115, the companion device receives the peripheral data from the wearable device. In step S1120, the companion device generates a result associated with the completion of a computing process by the companion device, the computing process being configured to use the peripheral data. In step S1125, the companion device communicates the result associated with the completion of the computing process to the wearable device.

[0084] Example 18. The method of example 17, wherein the wearable device can be smart glasses.

[0085] Example 19. The method of example 17, wherein the companion device can be at least one of another wearable device, a mobile device, a smartphone, a tablet, a server, and a device that includes a processor and an operating system.

[0086] Example 20. The method of example 17, further comprising: mirroring, by the companion device, data obtained from a peripheral device of the companion device on the wearable device as companion device peripheral data, by the companion device communicating, by the companion device, the companion device peripheral data to the companion device.

[0087] Example 21. The method of example 17, wherein the peripheral data may be inertial measurement unit (IMU) data, and the computing process may be a head pose manipulation, and completing the computing process by the wearable device may include using the result based on the head pose manipulation.

[0088] Example 22. The method of example 17, wherein the peripheral data may be image data, and the computing process may be a head pose manipulation, and completing the computing process by the wearable device may include using the result based on the head pose manipulation.

[0089] Example 23. The method of example 17, wherein the peripheral data may be image data, and the computing process may be an eye-tracking operation, and completing the computing process by the wearable device may include using the results based on the eye-tracking operation.

[0090] Example 24. The method of example 17, wherein the wearable device can include a first socket, the companion device can include a second socket communicatively coupled to the first socket, and mirroring the data associated with a peripheral device can include communicating the peripheral data between the first socket and the second socket.

[0091] Example 25. The method of example 17, wherein the companion device may include a virtual runtime environment, and a mirror associated with the peripheral device may be included in the virtual runtime environment.

[0092] Example 26. The method may include any combination of one or more of Examples 1-25.

[0093] Example 27. A non-transitory computer-readable storage medium, comprising instructions stored on the non-transitory computer-readable storage medium, the instructions, when executed by at least one processor, configured to cause a computing system to perform the method of any of Examples 1-26.

[0094] Example 28. An apparatus comprising means for carrying out the method according to any of Examples 1 to 26.

[0095] Example 29. An apparatus comprising at least one processor and at least one memory containing computer program code, the at least one memory and the computer program code configured to, using the at least one processor, cause the apparatus to perform at least the method of any of Examples 1 to 26.

[0096] Exemplary embodiments may include a non-transitory computer-readable storage medium including instructions that, when executed by at least one processor, are configured to cause a computing system to perform any of the methods described above. Exemplary embodiments may include an apparatus including means for performing any of the methods described above. Exemplary embodiments may include an apparatus including at least one processor and at least one memory including computer program code, the at least one memory and the computer program code configured to cause, with the at least one processor, to perform at least any of the methods described above.

[0097] Various implementations of the systems and techniques described herein may be realized in digital electronic circuitry, integrated circuits, specially designed ASICs (application specific integrated circuits), computer hardware, firmware, software, and / or combinations thereof. These various implementations may include implementation in one or more computer programs executable and / or interpretable by a programmable system including at least one programmable processor, which may be special-purpose or general-purpose, coupled to receive data and instructions from the storage system and to transmit data and instructions to the storage system.

[0098] These computer programs (also known as programs, software, software applications, or code) include machine instructions for a programmable processor and may be implemented in a high-level procedural and / or object-oriented programming language, and / or assembly / machine language. As used herein, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, apparatus, and / or device (e.g., magnetic disks, optical disks, memory, programmable logic circuits (PLDs)) used to provide machine instructions and / or data to a programmable processor, including a machine-readable medium that receives machine instructions as a machine-readable signal. The term "machine-readable signal" refers to any signal used to provide machine instructions and / or data to a programmable processor.

[0099] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having a display device (such as an LED (Light Emitting Diode) or OLED (Organic LED), or LCD (Liquid Crystal Display) monitor / screen) for displaying information to the user, as well as a keyboard and pointing device (e.g., a mouse or trackball) by which the user can provide input to the computer. Other types of devices can also be used to provide interaction with a user, for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback), and input from the user can be received in any form, including acoustic, spoken language, or tactile input.

[0100] The systems and techniques described herein may be implemented in a computing system that includes a back-end component (e.g., as a data server), or that includes a middleware component (e.g., an application server), or that includes a front-end component (e.g., a client computer having a graphical user interface or web browser through which a user can interact with an implementation of the systems and techniques described herein), or in a combination of such back-end, middleware, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communications network). Examples of communications networks include a local area network ("LAN"), a wide area network ("WAN"), and the Internet.

[0101] A computing system may include clients and servers. Clients and servers are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other.

[0102] Although several embodiments have been described, it will be understood that various modifications can be made without departing from the spirit and scope of the present specification.

[0103] Moreover, the logic flows depicted in the figures do not require the particular order shown, or sequential order, to achieve desired results. Moreover, other steps may be provided in or removed from the described flows, and other components may be added to or removed from the described systems. Accordingly, other implementations are within the scope of the following claims.

[0104] As described herein, while certain features of the described embodiments have been illustrated, numerous modifications, substitutions, changes, and equivalents will occur to those skilled in the art. It should therefore be understood that the appended claims are intended to cover all such modifications and changes that fall within the scope of the embodiments. They have been presented by way of example only, not limitation, and it should be understood that various changes in form and details may be made. Any part of the apparatus and / or method described herein may be combined in any combination, except in mutually exclusive combinations. The embodiments described herein may include various combinations and / or subcombinations of the functions, components, and / or features of the different embodiments described.

[0105] While exemplary embodiments may include various modifications and alternative forms, such embodiments are shown by way of example in the drawings and described in detail herein. It should be understood, however, that there is no intention to limit the exemplary embodiments to the particular forms disclosed, but on the contrary, the exemplary embodiments are to include all modifications, equivalents, and alternatives falling within the scope of the claims. Like numbers refer to like components throughout the description of the figures.

[0106] Some of the exemplary implementations described above are described as a process or method that is depicted as a flowchart. Although the flowcharts describe operations as sequential processes, many of the operations may occur in parallel, together, or simultaneously. Also, the order of operations may be rearranged. A process may be terminated when its operations are completed, or may have additional steps not included in the figures. A process may correspond to a method, a function, a procedure, a subroutine, a subprogram, etc.

[0107] The above methods, some of which are illustrated by flowcharts, may be implemented by hardware, software, firmware, middleware, microcode, hardware description languages, or any combination thereof. When implemented in software, firmware, middleware, or microcode, the program code or code segments to perform the necessary tasks may be stored in a machine-readable or computer-readable medium such as a storage medium. A processor(s) may perform the necessary tasks.

[0108] Specific structural and functional details disclosed herein are presented solely for purposes of describing exemplary embodiments, however, exemplary embodiments may be embodied in many alternative forms and should not be construed as being limited to only the embodiments set forth herein.

[0109] The terms first, second, etc. may be used herein to describe various elements, but it should be understood that these elements should not be limited by these terms. These terms are used only to distinguish one element from another. For example, a first element could be referred to as a second element, and similarly, a second element could be referred to as a first element, without departing from the scope of the exemplary embodiments. As used herein, the terms and / or include any and all combinations of one or more of the associated listed items.

[0110] When an element is referred to as being connected or coupled to another element, it should be understood that the element can be directly connected or coupled to the other element, or there may be intervening elements. In contrast, when an element is referred to as being directly connected or coupled to another element, there are no intervening elements. Other words used to describe the relationship between elements should be construed similarly (e.g., between vs. directly between, adjacent vs. directly adjacent, etc.).

[0111] The terminology used herein is for the purpose of describing particular embodiments and is not intended to limit the exemplary embodiments. As used herein, the singular forms a, an, and the are intended to include the plural forms as well, unless the context clearly indicates otherwise. It is further understood that the terms comprise, comprise / comprising, includes, and / or including, when used herein, specify the presence of stated features, integers, steps, operations, elements, and / or components, but do not preclude the presence or addition of one or more other features, components, steps, operations, elements, components, and / or groups thereof.

[0112] It should also be noted that in some alternative implementations, the functions / acts shown may occur out of the order noted in the figures. For example, two figures shown in succession may in fact be executed concurrently or the functions / acts may sometimes be executed in the reverse order, depending on the functions / acts involved.

[0113] Unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by those skilled in the art to which the exemplary embodiments belong. For example, terms such as those defined in commonly used dictionaries should be interpreted to have a meaning consistent with their meaning in the context of the relevant art, and should not be interpreted in an idealized or overly formal sense unless expressly defined in this specification.

[0114] The exemplary embodiments above and portions of the corresponding Detailed Description are presented in terms of software, or algorithms and symbolic representations of operations on data bits within a computer memory. These descriptions and representations are the ones that effectively convey the substance of their work to others skilled in the art. An algorithm, as the term is used here, and as it is used generally, is conceived to be a self-consistent sequence of steps leading to a desired result. The steps are those requiring physical manipulations of physical quantities. Usually, though not necessarily, these quantities take the form of optical, electrical, or magnetic signals capable of being stored, transferred, combined, compared, and otherwise manipulated. It has proven convenient at times, principally for reasons of common usage, to refer to these signals as bits, values, elements, symbols, characters, terms, numbers, or the like.

[0115] In the exemplary implementations described above, references to symbolic representations of acts and operations that may be implemented as program modules or functional processes (e.g., in the form of flowcharts) include routines, programs, objects, components, data structures, etc. that perform particular tasks or implement particular abstract data types, which may be described and / or implemented in existing structural elements and using existing hardware. Such existing hardware may include one or more central processing units (CPUs), digital signal processors (DSPs), application specific integrated circuits, field programmable gate array (FPGA) computers, etc.

[0116] It should be recognized, however, that all of these and similar terms are to be associated with the appropriate physical quantities and are merely convenient labels applied to these quantities. Unless otherwise indicated, or as will be apparent from the description, terms such as processing or computing or calculating or determining to display refer to actions and processes of a computer system or similar electronic computing device that manipulates data represented as physical quantities, electronic quantities in the computer system's registers and memory, and converts it into other data that is similarly represented as physical quantities in the computer system's memory or registers or other such information storage, transmission, or display devices.

[0117] It should also be noted that the software-implemented aspects of the exemplary implementations are typically encoded on some form of non-transitory program storage medium or implemented via some type of transmission medium. The program storage medium may be magnetic (e.g., a floppy disk or hard drive) or optical (e.g., a compact disk read-only memory, or CD ROM), and may be read-only or random access. Similarly, the transmission medium may be twisted wire pairs, coaxial cable, optical fiber, or some other suitable transmission medium known in the art. The exemplary implementations are not limited by these aspects of any given implementation.

[0118] Finally, it should also be noted that, although the appended claims set forth particular combinations of features described herein, the scope of the present disclosure is not limited to the specific combinations claimed below, but instead extends to encompass any combination of features of the embodiments disclosed herein, regardless of whether that particular combination is expressly recited in the appended claims at this time.

Claims

1. 1. A method comprising: Communicatively coupling a wearable device to a companion device; mirroring, by the wearable device, data acquired from a peripheral device of the wearable device on the companion device as peripheral data, wherein the mirroring comprises: acquiring, by the wearable device, the peripheral data from the peripheral device; communicating, by the wearable device, the peripheral data to the companion device; and the method further comprises: receiving, by the companion device, the peripheral data and processing the peripheral data into processed data; transmitting, by the companion device, the processed data to the wearable device; receiving, by the wearable device, the processed data and completing a computing process utilizing the processed data; A method comprising:

2. The method of claim 1 , further comprising: mirroring, by the wearable device, data obtained from a peripheral device of the companion device on the wearable device as companion device peripheral data, comprising receiving, by the wearable device, the companion device peripheral data from the companion device.

3. the peripheral data being inertial measurement unit (IMU) data; the computing process is a head pose operation; completing the computing process by the wearable device includes using a result of the head pose manipulation. The method according to claim 1 or claim 2.

4. The peripheral data is image data, the computing process is a head pose operation; completing the computing process by the wearable device includes using a result of the head pose manipulation. The method according to claim 1 or claim 2.

5. The peripheral data is image data, the computing process is an eye tracking operation; completing the computing process by the wearable device includes using results of the eye tracking operation. The method according to claim 1 or claim 2.

6. The wearable device includes a first socket; the companion device includes a second socket communicatively coupled to the first socket; mirroring the data associated with a peripheral device includes communicating the peripheral data between the first socket and the second socket. The method according to any one of claims 1 to 5.

7. The method according to any one of claims 1 to 6, wherein the wearable device is a smart glass.

8. The method according to any of claims 1 to 7, wherein the companion device is at least one of another wearable device, a mobile device, a smartphone, a tablet, a server, and a device including a processor and an operating system.

9. 1. A system comprising: Wearable devices and Companion devices and Equipped with The wearable device comprises: A device client; a hardware abstraction layer; an operating system abstraction layer; at least one peripheral device driver; Including, the companion device includes a runtime environment associated with the wearable device; The system comprises: and configured to mirror data associated with the peripheral device on the companion device as peripheral data, the mirroring comprising: acquiring, by the wearable device, the peripheral data from the peripheral device; communicating, by the wearable device, the peripheral data to the companion device; said system comprising: receiving, by the companion device, the peripheral data and processing the peripheral data into processed data; transmitting, by the companion device, the processed data to the wearable device; receiving, by the wearable device, the processed data and completing a computing process utilizing the processed data; It is configured as follows: system.

10. The system of claim 9 , wherein the wearable device is smart glasses.

11. The system of claim 9 or 10, wherein the companion device is at least one of another wearable device, a mobile device, a smartphone, a tablet, a server, and a device including a processor and an operating system.

12. The system of any one of claims 9 to 11, wherein mirroring by the wearable device data obtained from a peripheral device of the companion device as companion device peripheral data on the wearable device includes receiving by the wearable device the companion device peripheral data from the companion device.

13. the runtime environment is a virtual runtime environment that runs as a background process on the companion device; a mirror of the data associated with the peripheral device is included in the virtual runtime environment; A system according to any one of claims 9 to 12.

14. the peripheral data being inertial measurement unit (IMU) data; the computing process is a head pose operation; completing the computing process by the wearable device includes using a result generated based on the head pose manipulation. A system according to any one of claims 9 to 13.

15. The peripheral data is image data, the computing process is a head pose operation; completing the computing process by the wearable device includes using a result generated based on the head pose manipulation. A system according to any one of claims 9 to 13.

16. The peripheral data is image data, the computing process is an eye tracking operation; completing the computing process by the wearable device includes using results generated based on the eye tracking operations. A system according to any one of claims 9 to 13.

17. The wearable device includes a first socket; the companion device includes a second socket communicatively coupled to the first socket; mirroring the data associated with a peripheral device includes communicating the peripheral data between the first socket and the second socket. A system according to any one of claims 9 to 16.

18. 1. A method comprising: Communicatively coupling a wearable device to a companion device; mirroring, by the companion device, data associated with a peripheral device of the wearable device as peripheral data; receiving, by the companion device, the peripheral data from the wearable device; generating, by the companion device, a result associated with completion of a computing process by the companion device, the computing process being configured to use the peripheral data, the method further comprising: communicating, by the companion device, the result associated with the completion of the computing process to the wearable device; A method comprising:

19. The method of claim 18 , wherein the wearable device is smart glasses.

20. 20. The method of claim 18 or 19, wherein the companion device is at least one of another wearable device, a mobile device, a smartphone, a tablet, a server, and a device that includes a processor and an operating system.

21. A method according to any one of claims 18 to 20, further comprising: mirroring by the companion device on the wearable device data acquired from a peripheral device of the companion device as companion device peripheral data, the companion device communicating by the companion device the companion device peripheral data to the companion device.

22. the peripheral data being inertial measurement unit (IMU) data; the computing process is a head pose operation; completing the computing process by the wearable device includes using the results based on the head pose manipulation. A system according to any one of claims 18 to 21.

23. The peripheral data is image data, the computing process is a head pose operation; completing the computing process by the wearable device includes using the results based on the head pose manipulation. A method according to any one of claims 18 to 21.

24. The peripheral data is image data, the computing process is an eye tracking operation; completing the computing process by the wearable device includes using the results based on the eye tracking operations. A method according to any one of claims 18 to 21.

25. The wearable device includes a first socket; the companion device includes a second socket communicatively coupled to the first socket; mirroring the data associated with a peripheral device includes communicating the peripheral data between the first socket and the second socket. The method according to any one of claims 18 to 24.

26. the companion device includes a virtualization runtime environment; a mirror of the data associated with the peripheral device is included in the virtual runtime environment; The method according to any one of claims 18 to 25.

27. A non-transitory computer readable storage medium comprising instructions stored on the non-transitory computer readable storage medium, the instructions being configured, when executed by at least one processor, to cause a computing system to perform a method according to any of claims 1 to 8.

28. 27. A non-transitory computer readable storage medium comprising instructions stored on the non-transitory computer readable storage medium, the instructions being configured, when executed by at least one processor, to cause a computing system to perform a method according to any of claims 18 to 26.

29. Apparatus comprising means for carrying out the method according to any one of claims 1 to 8.

30. Apparatus comprising means for carrying out the method according to any of claims 18 to 26.

31. 1. An apparatus comprising: At least one processor; at least one memory containing computer program code; Equipped with The at least one memory and the computer program code are configured to cause the device, using the at least one processor, to perform at least the method according to any one of claims 1 to 8. Device.

32. 1. An apparatus comprising: At least one processor; at least one memory containing computer program code; Equipped with Apparatus, wherein said at least one memory and said computer program code are configured to cause said apparatus, using said at least one processor, to at least perform a method according to any of claims 18 to 26.

Images