Virtual screen using wearable device

Abstract

Systems, methods, and computer readable media for using a virtual screen of a wearable device are disclosed. An example method includes a computing device sending a definition of a virtual screen and screen data to an augmented reality (AR) head wearable device. The AR head-wearable device displays screen data on a display of the AR head-wearable device according to the definition of the virtual screen. The AR head wearable device displays a virtual user interface item applied to the AR head wearable device or the computing device to the user. The computing device identifies the user input as a user intent to be executed on the AR head wearable device and sends the user intent to the AR head wearable device. The computing device receives a location of the AR head-wearable device and adjusts screen data for the AR head-wearable device for display to the user according to the virtual screen definition.

Description

[0001] Priority Declaration

[0002] This application claims the benefit of U.S. Provisional Application No. 63 / 481,325, filed on January 24, 2023, the entire contents of which are hereby incorporated by reference. Technical Field

[0003] Embodiments of the present disclosure generally relate to using a virtual screen of a wearable device. More specifically, but not by way of limitation, embodiments of the present disclosure relate to a computing device sending virtual screen parameters and screen data to a virtual reality (VR), mixed reality (MR), or augmented reality (AR) head wearable device, wherein the VR, MR, or AR head wearable device presents the screen data to a user of the VR, MR, or AR head wearable device based on the virtual screen parameters. Background Art

[0004] Users increasingly expect virtual reality (VR), mixed reality (MR), and augmented reality (AR) wearable devices to operate in a more user-friendly manner while having more functionality. However, wearable devices typically have very little space for on-device interface controls, and AR head-worn devices typically have limited power to provide additional functionality. Furthermore, users increasingly desire more screen space to visualize their computer applications. BRIEF DESCRIPTION OF THE DRAWINGS

[0005] In the accompanying drawings (which are not necessarily drawn to scale), similar reference numerals may describe similar components in different views. To easily identify the discussion of any particular element or action, the most significant digit or digits in a reference numeral refer to the figure number in which the element is first introduced. Some non-limiting examples are shown in the figures of the accompanying drawings, in which:

[0006] Figure 1 is a diagrammatic representation of a networked environment in which the present disclosure may be deployed, according to some examples.

[0007] Figure 2 is a diagrammatic representation of a messaging system having both client-side and server-side functionality, according to some examples.

[0008] Figure 3 is a diagrammatic representation of data structures maintained in a database according to some examples.

[0009] Figure 4 is a diagrammatic representation of messages according to some examples.

[0010] Figure 5 A system of head wearable devices according to some examples is shown.

[0011] Figure 6 is a diagrammatic representation of a machine in the form of a computer system within which a set of instructions may be executed, causing the machine to perform any one or more of the methodologies discussed herein, according to some examples.

[0012] Figure 7 is a block diagram illustrating a software architecture within which examples may be implemented.

[0013] Figure 8 A system for monitor extension using a wearable device is shown according to some examples.

[0014] Figure 9 A system for monitor extension using a wearable device is shown according to some examples.

[0015] Figure 10 A user interface for configuring a virtual screen is shown in accordance with some embodiments.

[0016] Figure 11 A system with a virtual display is shown in accordance with some embodiments.

[0017] Figure 12 A method for using a virtual screen of a wearable device according to some embodiments is shown.

[0018] Figure 13 A method for using a virtual screen of a wearable device according to some embodiments is shown.

[0019] Figure 14 is a perspective view of a head-wearable device in the form of glasses, according to some examples. DETAILED DESCRIPTION

[0020] The following description includes the system, method, technology, instruction sequence and computer program product of the illustrative example of embodiment of the present disclosure.In the following description, for the purpose of illustration, set forth many specific details to provide the understanding of the various examples of the subject of the present invention.Yet, it will be apparent to those skilled in the art that the example of the subject of the present invention can be put into practice without these specific details.Usually, it is not necessary to show in detail known instruction examples, protocols, structures and technologies.

[0021] The term AR head wearable device is used as an illustrative device; however, those skilled in the art will recognize that the methods, systems, and computer-readable media disclosed herein are applicable to other wearable devices or non-wearable devices including VR devices, extended reality (XR) devices, and MR devices.

[0022] The technical challenge is how to provide additional screen space for computing devices. Users often want more screen space, but users are usually using mobile computing devices such as Figure 1 The mobile device 114 shown in FIG. 1 is not suitable for carrying an additional screen (e.g., a monitor or computer screen). Furthermore, additional screens are expensive, and providing power and space for additional screens can be a challenge. By using, for example, Figure 5 and Figure 14 The head wearable device 116 shown in FIG provides an additional screen (eg, monitor extension) to address this technical challenge. Figure 8 and Figure 9 , the computing device 114 sends the AV data 846 to the head wearable device 116, which displays the virtual screen at a position according to the defined position of the virtual screen. For example, Figure 11 The virtual screen B 1108 is defined as being offset above the computing device 114. The head wearable device 116 adjusts the AV data 846 based on the offset so that the virtual screen B 1108 appears to be in the defined position. The virtual screen B 1108 can display 3D graphics with objects appearing from the virtual screen.

[0023] Another technical challenge is how to provide additional user interface support to the user of the head-worn device 116, since the wearable device has very little space for interface controls. This technical challenge is addressed by mapping user interface items on the computing device 114 to the head-worn device 116. For example, a function key on the keyboard of the computing device 114 can be mapped to dim the display of the head-worn device 116. The computing device 114 determines that the user intends to perform an operation on the head-worn device 116 and sends an indication of the operation or the user's intention to the head-worn device 116.

[0024] Another technical challenge is how to provide user interface support for the combination of virtual screen and real screen. To solve this technical challenge, the head wearable device 116 displays user interface items that can be used for both real screen and virtual screen to the user. For example, Figure 11 , a virtual UI item 1102, shown as a virtual trash can, can be used by the user on both the display of the computing device 114 and the virtual screen B 1108. The head-worn device 116 performs hand tracking and is used to determine when the user selects a file on the virtual display or the real display and drags it to the virtual trash can, the virtual UI item 1102. The head-worn device 116 sends an instruction to the computing device 114 to carry out the user's intent. In this example, the user's intent is to delete the file.

[0025] Networked computing environment

[0026] Figure 1 1 is a block diagram illustrating an example interactive system 100 for facilitating interactions over a network (e.g., exchanging text messages, conducting text, audio, and video calls, or playing games). The interactive system 100 includes multiple client systems, each of which hosts multiple applications including an interactive client 104 and other applications 106. Each interactive client 104 is communicatively coupled to other instances of the interactive client 104 (e.g., hosted on respective other user systems), an interactive server system 110, and third-party servers 112 via one or more communication networks including a network 108 (e.g., the Internet). The interactive client 104 can also communicate with the locally hosted application 106 using an application programming interface (API).

[0027] Each user system 102 may include a plurality of user devices, such as a computing device 114 , a head wearable device 116 , and a computer client device 118 , which are communicatively coupled to exchange data and messages.

[0028] The interactive clients 104 interact with other interactive clients 104 and with the interactive server system 110 via the network 108. The data exchanged between the interactive clients 104 (e.g., interaction 120) and between the interactive clients 104 and the interactive server system 110 includes functions (e.g., commands for activating functions) and payload data (e.g., text, audio, video, or other multimedia data).

[0029] The interactive server system 110 provides server-side functionality to the interactive clients 104 via the network 108. Although certain functions of the interactive system 100 are described herein as being performed by either the interactive client 104 or the interactive server system 110, the location of certain functions within the interactive client 104 or within the interactive server system 110 may be a design choice. For example, it may be technically preferable to initially deploy certain technologies and functions within the interactive server system 110, but later migrate the technologies and functions to the interactive client 104 where the user system 102 has sufficient processing power.

[0030] The interactive server system 110 supports various services and operations provided to the interactive clients 104. Such operations include sending data to the interactive clients 104, receiving data from the interactive clients 104, and processing data generated by the interactive clients 104. The data may include message content, client device information, geographic location information, media enhancements and overlays, message content persistence conditions, social network information, and live event information. The data exchange within the interactive system 100 is activated and controlled by functions available through the user interface (UI) of the interactive clients 104.

[0031] Turning now specifically to the interaction server system 110, an application program interface (API) server 122 is coupled to the interaction server 124 and provides a programming interface thereto, making the functionality of the interaction server 124 accessible to the interaction clients 104, other applications 106, and third-party servers 112. The interaction server 124 is communicatively coupled to a database server 126, thereby facilitating access to a database 128 that stores data associated with interactions processed by the interaction server 124. Similarly, a web server 130 is coupled to the interaction server 124 and provides a web-based interface to the interaction server 124. To this end, the web server 130 processes incoming network requests via the Hypertext Transfer Protocol (HTTP) and several other related protocols.

[0032] The application program interface (API) server 122 receives and sends interaction data (e.g., commands and message payloads) between the interaction server 124 and client systems (e.g., interaction clients 104 and other applications 106), as well as third-party servers 112. Specifically, the application program interface (API) server 122 provides a set of interfaces (e.g., routines and protocols) that the interaction clients 104 and other applications 106 can call or query to activate the functionality of the interaction server 124. The application program interface (API) server 122 exposes various functions supported by the interaction server 124, including: account registration; login functionality; sending interaction data from a particular interaction client 104 to another interaction client 104 via the interaction server 124; transferring media files (e.g., images or videos) from the interaction client 104 to the interaction server 124; setting up a collection of media data (e.g., a story); retrieving a friend list of a user of the user system 102; retrieving messages and content; adding and removing entities (e.g., friends) from an entity graph (e.g., a social graph); locating friends in a social graph; and opening application events (e.g., related to the interaction client 104).

[0033] Interactive server 124 hosts multiple systems and subsystems, see below Figure 2 Provide a description.

[0034] Linked Applications

[0035] Returning to the interactive client 104, the features and functionality of an external resource (e.g., a linked application 106 or applet) are made available to the user via the interface of the interactive client 104. In this context, "external" refers to the fact that the application 106 or applet is external to the interactive client 104. External resources are typically provided by a third party, but may also be provided by the creator or provider of the interactive client 104. The interactive client 104 receives a user selection of an option to launch or access features of such an external resource. The external resource may be an application 106 installed on the user system 102 (e.g., a "local app"), or a small-scale version of an application (e.g., a "mini-program") hosted on the user system 102 or located remotely from the user system 102 (e.g., on a third-party server 112). The small-scale version of an application includes a subset of the features and functionality of the application (e.g., a full-scale local version of the application) and is implemented using a markup language document. In some examples, the small-scale version of an application (e.g., a "mini-program") is a web-based markup language version of the application and is embedded in the interactive client 104. In addition to using markup language documents (eg, .*ml files), applet may include scripting languages ​​(eg, .*js files or .json files) and style sheets (eg, .*ss files).

[0036] In response to receiving a user selection of an option to launch or access a feature of an external resource, the interactive client 104 determines whether the selected external resource is a web-based external resource or a locally installed application 106. In some cases, an application 106 locally installed on the user system 102 can be independent of and launched separately from the interactive client 104, for example, by selecting an icon corresponding to the application 106 on a home screen of the user system 102. A small-scale version of such an application can be launched or accessed via the interactive client 104, and in some examples, no portion of the small-scale application can be accessed outside of the interactive client 104 or only a limited portion of the small-scale application can be accessed outside of the interactive client 104. The small-scale application can be launched by the interactive client 104 receiving, for example, a markup language document associated with the small-scale application from a third-party server 112 and processing such a document.

[0037] In response to determining that the external resource is a locally installed application 106, the interactive client 104 instructs the user system 102 to launch the external resource by executing locally stored code corresponding to the external resource. In response to determining that the external resource is a web-based resource, the interactive client 104 communicates with the third-party server 112 (for example) to obtain a markup language document corresponding to the selected external resource. The interactive client 104 then processes the obtained markup language document to present the web-based external resource within the user interface of the interactive client 104.

[0038] The interactive client 104 can notify the user of the user system 102 or other users related to such user (e.g., "friends") of activities occurring in one or more external resources. For example, the interactive client 104 can provide participants in a conversation (e.g., a chat conversation) in the interactive client 104 with notifications related to external resources currently or recently used by one or more members of the user group. One or more users can be invited to join an active external resource or launch a recently used but currently inactive external resource (in a friend group). The external resource can provide participants in the conversation, each using a corresponding interactive client 104, with the ability to share items, conditions, states, or locations in the external resource with one or more members of the user group in a chat session. The shared items can be interactive chat cards that members of the chat can use to interact, for example, to launch the corresponding external resource, view specific information within the external resource, or take members of the chat to a specific location or state within the external resource. Within a given external resource, a response message can be sent to the user on the interactive client 104. The external resource can selectively include different media items in the response based on the current context of the external resource.

[0039] The interactive client 104 can present a list of available external resources (e.g., applications 106 or applets) to the user to launch or access a given external resource. The list can be presented in the form of a context-sensitive menu. For example, the icons representing different applications (or applets) of the application 106 (or applets) can change based on how the user launches the menu (e.g., from a conversational interface or from a non-conversational interface).

[0040] System Architecture

[0041] Figure 2 1 is a block diagram illustrating further details regarding the interactive system 100 according to some examples. Specifically, the interactive system 100 is shown to include an interactive client 104 and an interactive server 124. The interactive system 100 includes a plurality of subsystems that are supported on the client side by the interactive client 104 and on the server side by the interactive server 124. Example subsystems are discussed below.

[0042] Image processing system 202 provides various functions that enable a user to capture and enhance (eg, annotate or otherwise modify or edit) media content associated with a message.

[0043] The camera system 204 includes control software (e.g., in a camera application) that interacts with and controls the hardware camera hardware of the user system 102 (e.g., directly or via operating system control) to modify and enhance the real-time images captured and displayed via the interactive client 104.

[0044] The enhancement system 206 provides functionality related to the generation and publication of enhancements (e.g., media overlays) for images captured in real time by the camera of the user system 102 or retrieved from the memory of the user system 102. For example, the enhancement system 206 is operable to select, present, and display media overlays (e.g., image filters or image lenses) to the interactive client 104 for enhancing real-time images received via the camera system 204 or stored images retrieved from the memory 502 of the user system 102. These enhancements are selected and presented to the user of the interactive client 104 by the enhancement system 206 based on inputs and data, such as:

[0045] The geographic location of the user system 102; and

[0046] Social network information of users of the user system 102 .

[0047] The enhancement may include audio and visual content and visual effects. Examples of audio and visual content include pictures, text, logos, animations, and sound effects. Examples of visual effects include color overlays. Audio and visual content or visual effects may be applied to media content items (e.g., photos or videos) at user systems 102 for transmitting in messages, or may be applied to video content such as video content streams or feeds sent from interactive clients 104. Therefore, image processing system 202 may interact with and support various subsystems of communication system 208, such as messaging system 210 and video communication system 212.

[0048] Media overlays can include text or image data that can be superimposed on a photo taken by user system 102 or a video stream produced by user system 102. In some examples, the media overlay can be a location overlay (e.g., Venice Beach), the name of a live event, or a business name overlay (e.g., Beach Cafe). In other examples, image processing system 202 uses the geographic location of user system 102 to identify a media overlay that includes the name of a business at the geographic location of user system 102. The media overlay may include other tags associated with the business. The media overlay may be stored in database 128 and accessed by database server 126.

[0049] Image processing system 202 provides a user-based publishing platform that enables users to select a geographic location on a map and upload content associated with the selected geographic location. Users can also specify situations in which specific media overlays should be provided to other users. Image processing system 202 generates a media overlay that includes the uploaded content and associates it with the selected geographic location.

[0050] The augmented reality creation system 214 supports the augmented reality developer platform and includes applications for content creators (e.g., artists and developers) to create and publish augmentations (e.g., augmented reality experiences) for the interactive clients 104. The augmented reality creation system 214 provides content creators with a library of built-in features and tools, including, for example, custom shaders, tracking techniques, and templates.

[0051] In some examples, the enhancement creation system 214 provides a merchant-based publishing platform that enables merchants to select specific enhancements associated with a geographic location through a bidding process. For example, the enhancement creation system 214 associates the highest bidding merchant's media overlay with the corresponding geographic location for a predefined amount of time.

[0052] The communication system 208 is responsible for enabling and processing various forms of communication and interaction within the interactive system 100 and includes a messaging system 210, an audio communication system 216, and a video communication system 212. The messaging system 210 is responsible for enforcing temporary or time-limited access to content by the interactive clients 104. The messaging system 210 includes a plurality of timers (e.g., in a transient timer system 218) that selectively enable access (e.g., for presentation and display) of messages and associated content via the interactive clients 104 based on duration and display parameters associated with a message or a collection of messages (e.g., a story). Additional details regarding the operation of the transient timer system 218 are provided below. The audio communication system 216 enables and supports audio communication (e.g., real-time audio chat) between multiple interactive clients 104. Similarly, the video communication system 212 enables and supports video communication (e.g., real-time video chat) between multiple interactive clients 104.

[0053] The user management system 220 is operationally responsible for managing user data and profiles, and includes a social networking system 222 , which maintains information about relationships between users of the interactive system 100 .

[0054] The collection management system 224 is operationally responsible for managing collections or collections of media (e.g., collections of text, images, video, and audio data). Collections of content (e.g., messages, including images, video, text, and audio) can be organized into "event libraries" or "event stories." Such collections can be made available for a specified time period (e.g., the duration of the event to which the content relates). For example, content related to a concert can be made available as a "story" for the duration of the concert. The collection management system 224 is also responsible for publishing an icon to the user interface of the interactive client 104 that provides notification of a particular collection. The collection management system 224 includes curation functionality that enables collection managers to manage and curate specific content collections. For example, a curation interface enables event organizers to curate collections of content related to a specific event (e.g., removing inappropriate content or redundant messages). In addition, the collection management system 224 employs machine vision (or image recognition technology) and content rules to automatically curate content collections. In some examples, users can be compensated for including user-generated content in a collection. In such cases, the collection management system 224 operates to automatically pay such users for use of their content.

[0055] The mapping system 226 provides various geolocation functions and supports the presentation of map-based media content and messages by the interactive client 104. For example, the mapping system 226 enables the display of a user icon or avatar (e.g., stored in the profile data 302) on a map to indicate the current or past locations of the user's "friends" within the context of the map, as well as media content generated by these friends (e.g., a collection of messages including photos and videos). For example, on the map interface of the interactive client 104, a message posted by the user to the interactive system 100 from a particular geographic location can be displayed to the "friends" of a particular user within the context of a map of that particular location. A user can also share his or her location and status information with other users of the interactive system 100 via the interactive client 104 (e.g., using an appropriate status avatar), where the location and status information is similarly displayed to selected users within the context of the map interface of the interactive client 104.

[0056] The gaming system 228 provides various gaming functions within the context of the interactive client 104. The interactive client 104 provides a gaming interface that provides a list of available games that can be launched by a user within the context of the interactive client 104 and played with other users of the interactive system 100. The interactive system 100 also enables a particular user to invite other users to play a particular game by sending an invitation to such other users from the interactive client 104. The interactive client 104 also supports voice, video, and text messaging (e.g., chat) within the context of game play, provides leaderboards for games, and also supports the provision of in-game rewards (e.g., game coins and items).

[0057] The external resource system 230 provides an interface for the interactive client 104 to communicate with a remote server (e.g., a third-party server 112) to launch or access external resources (i.e., applications or applets). Each third-party server 112 hosts, for example, an application or a small-scale version of an application (e.g., a game application, a utility application, a payment application, or a ride-sharing application) based on a markup language (e.g., HTML5). The interactive client 104 can launch a web-based resource (e.g., an application) by accessing an HTML5 file from a third-party server 112 associated with the web-based resource. The applications hosted by the third-party server 112 are programmed in JavaScript using a software development kit (SDK) provided by the interactive server 124. The SDK includes an application program interface (API) with functions that can be called or activated by a web-based application. The interactive server 124 hosts a JavaScript library that provides access to a given external resource for specific user data of the interactive client 104. HTML5 is an example of a technology for programming games, but applications and resources programmed based on other technologies can be used.

[0058] To integrate the SDK's functionality into a web-based resource, the third-party server 112 downloads the SDK from the interaction server 124, or the third-party server 112 receives the SDK in some other manner. Once downloaded or received, the SDK is included as part of the application code of the web-based external resource. The code of the web-based resource can then call or activate certain functions of the SDK to integrate the features of the interaction client 104 into the web-based resource.

[0059] The SDK stored on the interactive server system 110 effectively provides a bridge between external resources (e.g., applications 106 or applets) and the interactive client 104. This gives the user a seamless experience of communicating with other users on the interactive client 104 while also preserving the appearance of the interactive client 104. In order to bridge the communication between the external resources and the interactive client 104, the SDK facilitates communication between the third-party server 112 and the interactive client 104. The WebViewJavaScriptBridge running on the user system 102 establishes two one-way communication channels between the external resources and the interactive client 104. Messages are sent asynchronously between the external resources and the interactive client 104 via these communication channels. Each SDK function activation is sent as a message and a callback. Each SDK function is implemented by constructing a unique callback identifier and sending a message with the callback identifier.

[0060] By using the SDK, not all information from the interactive client 104 is shared with the third-party server 112. The SDK limits which information is shared based on the needs of the external resource. Each third-party server 112 provides an HTML5 file corresponding to the web-based external resource to the interactive server 124. The interactive server 124 can add a visual representation of the web-based external resource (e.g., a box design or other graphics) in the interactive client 104. Once the user selects the visual representation or instructs the interactive client 104 to access a feature of the web-based external resource through the GUI of the interactive client 104, the interactive client 104 obtains the HTML5 file and instantiates the resource for accessing the feature of the web-based external resource.

[0061] The interactive client 104 presents a graphical user interface (e.g., a login page or title screen) for the external resource. During, before, or after presenting the login page or title screen, the interactive client 104 determines whether the launched external resource has previously been authorized to access the user data of the interactive client 104. In response to determining that the launched external resource has previously been authorized to access the user data of the interactive client 104, the interactive client 104 presents another graphical user interface of the external resource including the functions and features of the external resource. In response to determining that the launched external resource has not previously been authorized to access the user data of the interactive client 104, after displaying the login page or title screen of the external resource for a threshold period of time (e.g., 3 seconds), the interactive client 104 slides up a menu (e.g., animating the menu to emerge from the bottom of the screen to the middle or other portion of the screen) for authorizing the external resource to access the user data. The menu identifies the type of user data that the external resource is authorized to use. In response to receiving a user selection of the accept option, the interactive client 104 adds the external resource to the list of authorized external resources and allows the external resource to access the user data from the interactive client 104. External resources are authorized by the interactive client 104 to access user data under the OAuth 2 framework.

[0062] The interaction client 104 controls the type of user data shared with the external resource based on the type of external resource that is authorized. For example, an external resource comprising a full-scale application (e.g., application 106) is provided with access to a first type of user data (e.g., a two-dimensional avatar of the user with or without different avatar characteristics). As another example, an external resource comprising a small-scale version of an application (e.g., a web-based version of the application) is provided with access to a second type of user data (e.g., payment information, a two-dimensional avatar of the user, a three-dimensional avatar of the user, and an avatar with various avatar characteristics). Avatar characteristics include different ways to customize the appearance of an avatar (e.g., different poses, facial features, clothing, etc.).

[0063] The advertising system 232 operatively enables third parties to purchase advertisements for presentation to end users via the interactive clients 104 and also handles the delivery and presentation of these advertisements.

[0064] The virtual screen system 234 supports virtual screens for use with head wearable devices. Figure 8 System 800 and Figure 9The virtual screen system 234 receives the AV data 846 from the system 900 and processes the AV data 846 so that the virtual screen can be displayed at an appropriate viewing angle corresponding to the position of the virtual screen relative to the position of the head wearable device 116. In some embodiments, the virtual screen system 234 streams the AV data 846 to the head wearable device 116 as the AV data 846. For example, the virtual screen system 234 streams a movie to the head wearable device 116. In some embodiments, the virtual screen system 234 performs one or more functions described in conjunction with the systems 800 and 900.

[0065] Data Architecture

[0066] Figure 3 is a diagram illustrating a data structure 300 that may be stored in a database 304 of the interactive server system 110 according to certain examples. Although the contents of the database 304 are shown as including a plurality of tables, it should be understood that data may be stored in other types of data structures (e.g., an object-oriented database).

[0067] The database 304 includes message data stored in a message table 306. For any particular message, the message data includes at least message sender data, message recipient (or receiver) data, and payload. Figure 3 306 to describe additional details regarding the information that may be included in a message and within the message data stored in message table 306.

[0068] The entity table 308 stores entity data and is linked (e.g., by reference) to the entity graph 310 and the profile data 302. Entities for which records are maintained within the entity table 308 may include individuals, corporate entities, organizations, objects, places, events, and the like. Regardless of the entity type, any entity for which the interactive server system 110 stores data may be an identified entity. Each entity is provided with a unique identifier and an entity type identifier (not shown).

[0069] The entity graph 310 stores information about relationships and associations between entities. By way of example only, such relationships may be social, professional (e.g., working at a common company or organization), interest-based, or activity-based. Some relationships between entities may be one-way, such as a subscription by an individual user to digital content from a business or publication (e.g., a newspaper or other digital media channel or brand). Other relationships may be two-way, such as a "friend" relationship between individual users of the interactive system 100.

[0070] Certain permissions and relationships can be attached to each relationship, and can also be attached to each direction of the relationship. For example, a two-way relationship (e.g., a friend relationship between individual users) can include authorization to publish digital content items between the individual users, but can impose certain restrictions or filters on the publication of such digital content items (e.g., based on content characteristics, location data, or time of day data). Similarly, a subscription relationship between an individual user and a business user can impose varying degrees of restrictions on the publication of digital content from the business user to the individual user, and can significantly limit or prevent the publication of digital content from the individual user to the business user. As an example of an entity, a particular user can record certain restrictions in the record for that entity within the entity table 308 (e.g., through privacy settings). Such privacy settings can apply to all types of relationships in the context of the interactive system 100, or can be selectively applied to certain types of relationships.

[0071] Profile data 302 stores various types of profile data about a particular entity. Based on the privacy settings specified by the particular entity, profile data 302 can be selectively used and presented to other users of the interactive system 100. In the case where the entity is a person, profile data 302 includes, for example, the user's name, phone number, address, settings (e.g., notification and privacy settings), and an avatar representation (or a collection of such avatar representations) selected by the user. The particular user can then selectively include one or more of these avatar representations within the content of messages transmitted via the interactive system 100 and on a map interface displayed to other users by the interactive client 104. The collection of avatar representations can include a "status avatar," which presents a graphical representation of the user's status or activity that they may choose to transmit at a particular time.

[0072] Where the entity is a group, the profile data 302 for the group may similarly include one or more avatar representations associated with the group, in addition to the group name, members, and various settings for the relevant group (eg, notifications).

[0073] Database 304 also stores enhancement data, such as overlays or filters, in enhancement table 312. Enhancement data is associated with and applied to videos (video data is stored in video table 314) and images (image data is stored in image table 316).

[0074] In some examples, a filter is displayed as an overlay on an image or video during presentation to a recipient user. Filters can be of various types, including filters selected by the user from a set of filters that the interactive client 104 presents to the sending user while the sending user is composing a message. Other types of filters include geolocation filters (also known as geofilters), which can be presented to the sending user based on geographic location. For example, a geolocation filter specific to a nearby or special location can be presented within the user interface by the interactive client 104 based on geographic location information determined by a global positioning system (GPS) unit of the user system 102.

[0075] Another type of filter is a data filter, which can be selectively presented to the sending user by the interaction client 104 based on other input or information collected during the message creation process by the user system 102. Examples of data filters include the current temperature at a particular location, the current speed the sending user is traveling, the battery life of the user system 102, or the current time.

[0076] Other augmented data that can be stored in the image table 316 include augmented reality content items (e.g., corresponding to application "lenses" or augmented reality experiences). Augmented reality content items can be real-time special effects and sounds that can be added to images or videos.

[0077] The story table 318 stores data about a collection of messages and associated images, video, or audio data that are compiled into a collection (e.g., a story or gallery). The creation of a particular collection can be initiated by a particular user (e.g., each user for whom a record is maintained in the entity table 308). A user can create a "personal story" in the form of a collection of content that has been created and sent / broadcasted by the user. To this end, the user interface of the interaction client 104 can include a user-selectable icon that enables the sending user to add specific content to his or her personal story.

[0078] The collection can also constitute a "live story", which is a collection of content from multiple users created manually, automatically, or using a combination of manual and automatic techniques. For example, a "live story" can constitute a curated stream of user-submitted content from different locations and events. Users whose client devices have location services enabled and who are at a common location event at a particular time can be presented with the option to contribute content to a particular live story, for example, via the user interface of the interactive client 104. Live stories can be identified to the user by the interactive client 104 based on his or her location. The end result is a "live story" told from the perspective of the group.

[0079] Another type of content collection is called a "location story," which enables users whose user systems 102 are located in a particular geographic location (e.g., on a college or university campus) to contribute to a particular collection. In some examples, contributions to location stories may employ secondary authentication to verify that the end user belongs to a particular organization or other entity (e.g., is a student on a university campus).

[0080] As mentioned above, video table 314 stores video data that, in some examples, is associated with messages for which records are maintained within message table 306. Similarly, image table 316 stores image data associated with messages whose message data is stored in entity table 308. Entity table 308 can associate various enhancements from enhancement table 312 with the various images and videos stored in image table 316 and video table 314.

[0081] The database 304 also includes a virtual screen table 318. The virtual screen table 318 includes virtual screen data for streaming to the head wearable device 116 and other data supporting the virtual screen system 234.

[0082] Data communication architecture

[0083] Figure 4 is a schematic diagram illustrating the structure of a message 400 according to some examples, the message 400 being generated by an interaction client 104 for transmission to another interaction client 104 via an interaction server 124. The content of a particular message 400 is used to populate a message table 306 stored within a database 304 accessible by the interaction server 124. Similarly, the content of the message 400 is stored in memory as "in-transit" or "in-flight" data of the user system 102 or the interaction server 124. The message 400 is shown as including the following example components:

[0084] Message identifier 402 is a unique identifier that identifies message 400 .

[0085] Message text payload 404 is the text to be generated by the user via the user interface of user system 102 and included in message 400 .

[0086] • Message image payload 406 is image data captured by the camera component of user system 102 or retrieved from the memory component of user system 102 and included in message 400. Image data for a message 400 sent or received may be stored in image table 316.

[0087] • Message video payload 408: Video data captured by the camera component or retrieved from the memory component of the user system 102 and included in the message 400. The video data for a message 400 sent or received may be stored in the image table 316.

[0088] • Message audio payload 410 : audio data captured by a microphone or retrieved from a memory component of the user system 102 and included in the message 400 .

[0089] Message enhancement data 412: Enhancement data (e.g., filters, stickers, or other annotations or enhancements) representing enhancements to be applied to the message image payload 406, message video payload 408, or message audio payload 410 of the message 400. Enhancement data for a sent or received message 400 may be stored in the enhancement table 312.

[0090] The message duration parameter 414 is a parameter value indicating the amount of time, in seconds, that the content of the message (e.g., message image payload 406, message video payload 408, message audio payload 410) is to be presented to or made accessible to the user via the interactive client 104.

[0091] Message geolocation parameters 416: Geolocation data (e.g., latitude and longitude coordinates) associated with the content payload of the message. Multiple message geolocation parameter 416 values ​​may be included in the payload, with each of these parameter values ​​being associated with a content item included in the content (e.g., a specific image within the message image payload 406 or a specific video within the message video payload 408).

[0092] Message story identifier 418: An identifier value that identifies one or more content collections (e.g., a "story" identified in stories table 318) associated with a particular content item in message image payload 406 of message 400. For example, the identifier value can be used to associate multiple images within message image payload 406 with multiple content collections.

[0093] Message tags 420: Each message 400 can be tagged with a plurality of tags, each of which indicates the subject matter of the content included in the message payload. For example, if a particular image included in the message image payload 406 depicts an animal (e.g., a lion), a tag value can be included within the message tags 420 indicating the relevant animal. The tag value can be manually generated based on user input, or can be automatically generated using, for example, image recognition.

[0094] • Message sender identifier 422: An identifier (eg, a messaging system identifier, an email address, or a device identifier) ​​that indicates the user of the user system 102 on which the message 400 was generated and from which the message 400 was sent.

[0095] • Message recipient identifier 424: Indicates an identifier (eg, a messaging system identifier, email address, or device identifier) ​​of the user of user system 102 to which message 400 is addressed.

[0096] The content (e.g., value) of each component of message 400 may be a pointer to a location in a table where the content data value is stored. For example, the image value in message image payload 406 may be a pointer to a location (or its address) within image table 316. Similarly, the value within message video payload 408 may point to data stored within image table 316, the value stored within message enhancement data 412 may point to data stored in enhancement table 312, the value stored within message story identifier 418 may point to data stored in story table 318, and the values ​​stored within message sender identifier 422 and message recipient identifier 424 may point to user records stored in entity table 308.

[0097] System with head wearable device

[0098] Figure 5 A system 500 is shown that includes a head wearable device 116 having a selector input device according to some examples. Figure 5 is a high-level functional block diagram of an example head wearable device 116 communicatively coupled to a computing device 114 and various server systems 504 (eg, interactive server system 110 ) via various networks 108 .

[0099] The head wearable device 116 includes one or more cameras, each of which may be, for example, a visible light camera 506 , an infrared emitter 508 , and an infrared camera 510 .

[0100] Computing device 114 is connected to head wearable device 116 using both low-power wireless connection 512 and high-speed wireless connection 514. According to some examples, computing device 114 is also connected to server system 504 and network 516. Computing device 114 can be a portable computing device such as a smartphone, a tablet computer, a laptop computer, or another type of computing device 114 such as a desktop computer, or another type of computing device 114.

[0101] The head wearable device 116 also includes two image displays in the optical assembly's image displays 518. The two optical assembly image displays 518 include an image display associated with the left lateral side of the head wearable device 116 and an image display associated with the right lateral side of the head wearable device 116. The head wearable device 116 also includes an image display driver 520, an image processor 522, low power circuitry 524, and high speed circuitry 526. The optical assembly's image displays 518 are used to present images and videos, including images that may include a graphical user interface, to a user of the head wearable device 116.

[0102] The image display driver 520 commands and controls the image display 518 of the optical assembly. The image display driver 520 can deliver image data directly to the image display 518 of the optical assembly for presentation or can convert the image data into a signal or data format suitable for delivery to an image display device. For example, the image data can be video data formatted according to a compression format such as H.264 (MPEG-4 Part 10), HEVC, Theora, Dirac, RealVideo RV40, VP8, VP9, ​​etc., while the still image data can be formatted according to a compression format such as Portable Network Graphics (PNG), Joint Photographic Experts Group (JPEG), Tagged Image File Format (TIFF), or Exchangeable Image File Format (EXIF).

[0103] The head-wearable device 116 includes a frame and stems (or temples) extending from lateral sides of the frame. The head-wearable device 116 also includes a user input device 528 (e.g., a touch sensor or a push button), comprising an input surface on the head-wearable device 116. The user input device 528 (e.g., a touch sensor or a push button) is used to receive input selections from the user for manipulating the graphical user interface of the presented image.

[0104] Figure 5 The components of the head wearable device 116 shown in FIG are located on one or more circuit boards (e.g., PCBs or flexible PCBs) in the frame or temples. Alternatively or additionally, the depicted components may be located in chunks, frames, hinges, or nosepieces of the head wearable device 116. The left and right visible light cameras 506 may include digital camera elements, such as complementary metal oxide semiconductor (CMOS) image sensors, charge coupled devices, camera lenses, or any other corresponding visible light or light capturing elements that can be used to capture data, including images of scenes with unknown objects.

[0105] The head wearable device 116 includes a memory 502 that stores instructions for performing a subset or all of the functions described herein. The memory 502 may also include a storage device.

[0106] like Figure 5 As shown in FIG, high-speed circuitry 526 includes a high-speed processor 530, memory 502, and high-speed wireless circuitry 532. In some examples, image display driver 520 is coupled to high-speed circuitry 526 and is operated by high-speed processor 530 to drive the left and right image displays in image display 518 of the optical assembly. High-speed processor 530 can be any processor capable of managing high-speed communications and operations of any general-purpose computing system required for head wearable device 116. High-speed processor 530 includes the processing resources required to manage high-speed data transmission over high-speed wireless connection 514 to a wireless local area network (WLAN) using high-speed wireless circuitry 532. In some examples, high-speed processor 530 executes an operating system (e.g., a LINUX operating system) or other such operating system for head wearable device 116, and the operating system is stored in memory 502 for execution. In addition to any other responsibilities, high-speed processor 530, which executes the software architecture of head wearable device 116, manages data transmission with high-speed wireless circuitry 532. In some examples, high-speed wireless circuitry 532 is configured to implement the Institute of Electrical and Electronics Engineers (IEEE) 802.11 communication standard, also referred to herein as WiFi. In some examples, high-speed wireless circuitry 532 can implement other high-speed communication standards.

[0107] The low power wireless circuit 534 and the high speed wireless circuit 532 of the head wearable device 116 may include a short range transceiver (Bluetooth TM ) and a wireless wide area network transceiver, a wireless local area network transceiver, or a wide area network transceiver (e.g., cellular or WiFi). The computing device 114, including the transceivers for communicating via the low power wireless connection 512 and the high speed wireless connection 514, can be implemented using details of the architecture of the head wearable device 116, as can other elements of the network 516.

[0108] Memory 502 comprises any storage device capable of storing various data and applications, including camera data generated by left and right visible light cameras 506, infrared camera 510, and image processor 522, as well as images generated for display on an image display 518 of the optical assembly via image display driver 520. Although memory 502 is shown integrated with high-speed circuitry 526, in some examples, memory 502 may be a separate, standalone component of head-wearable device 116. In some such examples, electrical wiring may provide a connection from image processor 522 or low-power processor 536 to memory 502 via a chip including high-speed processor 530. In some examples, high-speed processor 530 may manage addressing of memory 502, such that low-power processor 536 will initiate high-speed processor 530 whenever a read or write operation involving memory 502 is required.

[0109] like Figure 5 As shown, the low-power processor 536 or the high-speed processor 530 of the head wearable device 116 can be coupled to a camera device (a visible light camera device 506, an infrared emitter 508 or an infrared camera device 510), an image display driver 520, a user input device 528 (for example, a touch sensor or a push button) and a memory 502.

[0110] The head-worn device 116 is connected to a host computer. For example, the head-worn device 116 is paired with the computing device 114 via a high-speed wireless connection 514 or is connected to a server system 504 via a network 516. The server system 504 can be one or more computing devices that are part of a service or network computing system, for example, including a processor, memory, and a network communication interface to communicate with the computing device 114 and the head-worn device 116 via the network 516.

[0111] The computing device 114 includes a processor and a network communication interface coupled to the processor. The network communication interface allows communication via a network 516, a low-power wireless connection 512, or a high-speed wireless connection 514. The computing device 114 may also store at least a portion of the instructions for generating binaural audio content in a memory of the computing device 114 to implement the functionality described herein.

[0112] The output components of the head wearable device 116 include visual components, such as a display (e.g., a liquid crystal display (LCD), a plasma display panel (PDP), a light emitting diode (LED) display, a projector, or a waveguide). The image display of the optical assembly is driven by the image display driver 520. The output components of the head wearable device 116 also include acoustic components (e.g., speakers), tactile components (e.g., vibration motors), other signal generators, etc. The input components of the head wearable device 116, the computing device 114, and the server system 504 (e.g., user input devices 528) may include alphanumeric input components (e.g., a keyboard, a touch screen configured to receive alphanumeric input, an optical keyboard, or other alphanumeric input components), point-based input components (e.g., a mouse, touchpad, trackball, joystick, motion sensor, or other pointing instrument), tactile input components (e.g., physical buttons, a touch screen or other tactile input components that provide the location and force of a touch or touch gesture), audio input components (e.g., a microphone), etc.

[0113] The head wearable device 116 may also include additional peripheral elements. Such peripheral elements may include biometric sensors, additional sensors, or display elements integrated with the head wearable device 116. For example, the peripheral elements may include any I / O components, including output components, motion components, positioning components, or any other such components described herein.

[0114] For example, the biometric component includes a component for detecting expressions (e.g., hand expressions, facial expressions, voice expressions, body postures, or eye tracking), measuring biological signals (e.g., blood pressure, heart rate, body temperature, sweating, or brain waves), identifying people (e.g., voice recognition, retinal recognition, facial recognition, fingerprint recognition, or electroencephalogram-based recognition), etc. The motion component includes an acceleration sensor component (e.g., an accelerometer), a gravity sensor component, a rotation sensor component (e.g., a gyroscope), etc. The positioning component includes a position sensor component for generating position coordinates (e.g., a global positioning system (GPS) receiver component), a Wi-Fi or Bluetooth receiver for generating positioning system coordinates, etc. TM transceiver, altitude sensor components (e.g., an altimeter or barometer that detects air pressure, from which altitude can be derived), orientation sensor components (e.g., a magnetometer), etc. Such positioning system coordinates can also be received from the computing device 114 via the low-power wireless connection 512 and the high-speed wireless connection 514 via the low-power wireless circuit 534 or the high-speed wireless circuit 532.

[0115] Machine Architecture

[0116] Figure 66 is a diagrammatic representation of a machine 600 within which instructions 602 (e.g., software, programs, applications, applet, apps, or other executable code) may be executed for causing the machine 600 to perform any one or more of the methodologies discussed herein. For example, the instructions 602 may cause the machine 600 to perform any one or more of the methodologies described herein. The instructions 602 transform a general-purpose, unprogrammed machine 600 into a specialized machine 600 that is programmed to perform the functions described and illustrated in the manner described. The machine 600 may operate as a standalone device or may be coupled (e.g., networked) to other machines. In a networked deployment, the machine 600 may operate as a server or a client machine in a server-client network environment, or as a peer machine in a peer-to-peer (or distributed) network environment. The machine 600 may include, but is not limited to, a server computer, a client computer, a personal computer (PC), a tablet computer, a laptop computer, a netbook, a set-top box (STB), a personal digital assistant (PDA), an entertainment media system, a cellular phone, a smartphone, a mobile device, a wearable device (e.g., a smart watch), a smart home device (e.g., a smart appliance), other smart devices, a web device, a network router, a network switch, a network bridge, or any machine capable of sequentially or otherwise executing instructions 602 specifying actions to be taken by the machine 600. In addition, although only a single machine 600 is shown, the term "machine" should also be construed to include a collection of machines that individually or jointly execute instructions 602 to perform any one or more of the methods discussed herein. For example, the machine 600 may include the user system 102 or any of a plurality of server devices forming part of the interactive server system 110. In some examples, the machine 600 may also include both a client system and a server system, wherein certain operations of a particular method or algorithm are performed on the server side and certain operations of the particular method or algorithm are performed on the client side.

[0117] The machine 600 may include a processor 604, a memory 606, and input / output I / O components 608 that may be configured to communicate with each other via a bus 610. In an example, the processor 604 (e.g., a central processing unit (CPU), a reduced instruction set computing (RISC) processor, a complex instruction set computing (CISC) processor, a graphics processing unit (GPU), a digital signal processor (DSP), an application specific integrated circuit (ASIC), a radio frequency integrated circuit (RFIC), another processor, or any suitable combination thereof) may include, for example, a processor 612 that executes instructions 602 and a processor 614. The term "processor" is intended to include multi-core processors, which may include two or more independent processors (sometimes referred to as "cores") that may execute instructions concurrently. Although Figure 6 Multiple processors 604 are shown, but the machine 600 may include a single processor with a single core, a single processor with multiple cores (e.g., a multi-core processor), multiple processors with a single core, multiple processors with multiple cores, or any combination thereof.

[0118] The memory 606 includes a main memory 616, a static memory 618, and a storage unit 620, all of which are accessible by the processor 604 via the bus 610. The main memory 606, the static memory 618, and the storage unit 620 store instructions 602 that implement any one or more of the methodologies or functions described herein. The instructions 602 may also reside, completely or partially, within the main memory 616, within the static memory 618, within the machine-readable medium 622 within the storage unit 620, within at least one of the processors 604 (e.g., within a cache memory of a processor), or any suitable combination thereof during execution by the machine 600.

[0119] The I / O components 608 may include various components for receiving input, providing output, generating output, sending information, exchanging information, capturing measurements, etc. The specific I / O components 608 included in a particular machine will depend on the type of machine. For example, a portable machine such as a mobile phone may include a touch input device or other such input mechanism, while a headless server machine will be less likely to include such a touch input device. It should be appreciated that the I / O components 608 may include Figure 6 Many other components are not shown in the drawings. In various examples, the I / O components 608 may include user output components 624 and user input components 626. The user output components 624 may include visual components (e.g., displays such as plasma display panels (PDPs), light emitting diode (LED) displays, liquid crystal displays (LCDs), projectors, or cathode ray tubes (CRTs)), acoustic components (e.g., speakers), tactile components (e.g., vibration motors, resistance mechanisms), other signal generators, etc. The user input components 626 may include alphanumeric input components (e.g., keyboards, touch screens configured to receive alphanumeric input, optical keyboards, or other alphanumeric input components), point-based input components (e.g., mice, touch pads, trackballs, joysticks, motion sensors, or other pointing instruments), tactile input components (e.g., physical buttons, touch screens or other tactile input components that provide location and force of touches or touch gestures), audio input components (e.g., microphones), etc.

[0120] In other examples, the I / O component 608 may include a biometric component 628, a motion component 630, an environmental component 632, or a positioning component 634, as well as a wide range of other components. For example, the biometric component 628 includes components for detecting expressions (e.g., hand expressions, facial expressions, voice expressions, body postures, or eye tracking), measuring biosignals (e.g., blood pressure, heart rate, body temperature, sweat, or brain waves), identifying people (e.g., voice recognition, retinal recognition, facial recognition, fingerprint recognition, or electroencephalogram-based recognition), etc. The motion component 630 includes an acceleration sensor component (e.g., an accelerometer), a gravity sensor component, and a rotation sensor component (e.g., a gyroscope).

[0121] The environmental components 632 include, for example, one or more cameras (with still image / photo and video capabilities), an illumination sensor component (e.g., a photometer), a temperature sensor component (e.g., one or more thermometers that detect ambient temperature), a humidity sensor component, a pressure sensor component (e.g., a barometer), an acoustic sensor component (e.g., one or more microphones that detect background noise), a proximity sensor component (e.g., an infrared sensor that detects nearby objects), a gas sensor (e.g., a gas detection sensor for detecting concentrations of hazardous gases for safety purposes or for measuring pollutants in the atmosphere), or other components that can provide indications, measurements, or signals corresponding to the surrounding physical environment.

[0122] With respect to cameras, the user system 102 can have a camera system that includes, for example, a front-facing camera on the front surface of the user system 102 and a rear-facing camera on the rear surface of the user system 102. The front-facing camera can, for example, be used to capture still images and videos of the user of the user system 102 (e.g., “selfies”), which can then be enhanced with the enhancement data (e.g., filters) described above. The rear-facing camera can, for example, be used to capture still images and videos in a more traditional camera mode, which are similarly enhanced with the enhancement data. In addition to the front-facing camera and the rear-facing camera, the user system 102 can also include a 360° camera for capturing 360° photos and videos.

[0123] Furthermore, the camera system of the user system 102 may include dual rear cameras (e.g., a main camera and a depth-sensing camera), or even triple, quad, or quintuple rear camera configurations on the front and back sides of the user system 102. For example, these multi-camera systems may include a wide-angle camera, an ultra-wide-angle camera, a telephoto camera, a macro camera, and a depth sensor.

[0124] The positioning component 634 includes a position sensor component (for example, a GPS receiver component), an altitude sensor component (for example, an altimeter or barometer that detects air pressure, and the altitude can be obtained according to the air pressure), an orientation sensor component (for example, a magnetometer), etc.

[0125] Various technologies can be used to achieve communication. The I / O components 608 also include a communication component 636 that is operable to couple the machine 600 to a network 638 or device 640 via corresponding couplings or connections. For example, the communication component 636 may include a network interface component or other suitable device that interfaces with the network 638. In other examples, the communication component 636 may include a wired communication component, a wireless communication component, a cellular communication component, a near field communication (NFC) component, Components (e.g. Low energy consumption), Device 640 may be another machine or any of a variety of peripheral devices (eg, a peripheral device coupled via USB).

[0126] In addition, the communication component 636 can detect an identifier, or include a component operable to detect an identifier. For example, the communication component 636 can include a radio frequency identification (RFID) tag reader component, an NFC smart tag detection component, an optical reader component (e.g., an optical sensor for detecting one-dimensional bar codes such as Universal Product Code (UPC) bar codes, multi-dimensional bar codes such as Quick Response (QR) codes, Aztec codes, Data Matrix, Dataglyph, MaxiCode, PDF417, UltraCode, UCC RSS-2D bar codes, and other optical codes), or an acoustic detection component (e.g., a microphone for identifying an audio signal of a tag). In addition, various information can be obtained via the communication component 636, such as location via Internet Protocol (IP), geolocation via Internet Protocol (IP), location information ... Position derived from signal triangulation, position derived via detection of NFC beacon signals that can indicate a specific location, etc.

[0127] Various memories (e.g., main memory 616, static memory 618, and memory of processor 604) and storage unit 620 may store one or more sets of instructions and data structures (e.g., software) implemented or used by any one or more of the methods or functions described herein. These instructions (e.g., instructions 602) when executed by processor 604 cause various operations to implement the disclosed examples.

[0128] Instructions 602 may be sent or received over network 638 using a transmission medium via a network interface device (e.g., a network interface component included in communications component 636) and using any of several well-known transmission protocols (e.g., Hypertext Transfer Protocol (HTTP)). Similarly, instructions 602 may be sent or received via a coupling (e.g., a peer-to-peer coupling) with device 640 using a transmission medium.

[0129] Software Architecture

[0130] Figure 7 7 is a block diagram 700 illustrating a software architecture 702 that can be installed on any one or more of the devices described herein. The software architecture 702 is supported by hardware, such as a machine 704 including a processor 706, memory 708, and I / O components 710. In this example, the software architecture 702 can be conceptualized as a stack of layers, where each layer provides specific functionality. The software architecture 702 includes layers such as an operating system 712, libraries 714, frameworks 716, and applications 718. In operation, the applications 718 invoke API calls 720 through the software stack and receive messages 722 in response to the API calls 720.

[0131] The operating system 712 manages hardware resources and provides common services. The operating system 712 includes, for example, a kernel 724, services 726, and drivers 728. The kernel 724 serves as an abstraction layer between the hardware and other software layers. For example, the kernel 724 provides functions such as memory management, processor management (e.g., scheduling), component management, networking, and security settings. Services 726 can provide other common services to other software layers. Drivers 728 are responsible for controlling or interfacing with the underlying hardware. For example, drivers 728 may include display drivers, camera drivers, or Low-power drivers, Flash drivers, serial communication drivers (e.g., USB drivers), drivers, audio drivers, power management drivers, etc.

[0132] The libraries 714 provide a common low-level infrastructure used by the applications 718. The libraries 714 may include system libraries 730 (e.g., C standard libraries) that provide functions such as memory allocation functions, string manipulation functions, mathematical functions, etc. In addition, the libraries 714 may include API libraries 732, such as media libraries (e.g., libraries for supporting the presentation and manipulation of various media formats, such as Moving Picture Experts Group-4 (MPEG4), Advanced Video Coding (H.264 or AVC), Moving Picture Experts Group Layer-3 (MP3), Advanced Audio Coding (AAC), Adaptive Multi-Rate (AMR) audio codec, Joint Photographic Experts Group (JPEG or JPG), or Portable Network Graphics (PNG)), graphics libraries (e.g., the OpenGL framework for rendering graphical content on a display in two dimensions (2D) and three dimensions (3D), database libraries (e.g., SQLite providing various relational database functions), web libraries (e.g., WebKit providing web browsing functions), etc. The libraries 714 may also include a variety of other libraries 734 to provide many other APIs to the applications 718.

[0133] The framework 716 provides a common high-level infrastructure used by applications 718. For example, the framework 716 provides various graphical user interface (GUI) functions, high-level resource management, and high-level location services. The framework 716 can provide a wide range of other APIs that can be used by applications 718, some of which may be specific to a particular operating system or platform.

[0134] In an example, applications 718 may include a home application 736, a contacts application 738, a browser application 740, a book reader application 742, a location application 744, a media application 746, a messaging application 748, a game application 750, and a variety of other applications such as third-party applications 752. Applications 718 are programs that perform functions defined in the program. Various programming languages ​​may be used to create one or more of the applications 718 structured in various ways, such as an object-oriented programming language (e.g., Objective-C, Java, or C++) or a procedural programming language (e.g., C or assembly language). In a specific example, third-party applications 752 (e.g., those developed by entities other than the vendor of a particular platform using ANDROID) may be used to create a third-party application 752. TM or IOS TM Software Development Kit (SDK) can be used to develop applications on platforms such as IOS TM ANDROID TM 、 Mobile software running on the mobile operating system of the phone or other mobile operating system. In this example, third-party applications 752 can activate API calls 720 provided by the operating system 712 to facilitate the functions described in this article.

[0135] Using virtual screens on wearable devices

[0136] Figure 8 Shown is a system 800 for monitor extension using a wearable device, according to some examples. Figure 8 Shown as Figure 1 and Figure 5 The head wearable device 116 is shown in FIG. The head wearable device 116 communicates with the head wearable device 116 via communication 852, 854, 856, 858 and 859. Figure 9 114 to communicate with the computing device 114.

[0137] The real-time service 832 communicates with the desktop connection server 920 of the computing device 114. Communications 852, 854, 856, 858, and 859 are via wired, wireless, or via a network 860, which can be the Internet or another network. In some embodiments, communications 852, 854, 856, 858, and 859 are encoded for security. According to some examples, the real-time server 832 uses a utility (util) server 848 to determine an IP address or other address to establish a peer-to-peer connection with the computing device 114. According to some examples, the utility server 848 is a Session Traversal Utilities for NAT (STUN) server and a Traversal Using Relays around NAT (TURN) server.

[0138] The control channel 924 is a channel for the real-time service 832 to exchange control information, such as input / output (IO) data 847, with the computing device 114 via communication 858 or via communication 858. IO data 847 includes information about input from the computing device 114 to the head wearable device 116 and input from the head wearable device 116 to the computing device 114. The media channel 828 is a channel for the real-time service 832 to receive audio / video (AV) data 846 from the computing device 114 via communication 856 or communication 859. According to some embodiments, AV data 846 includes timestamps and video or screen data. The control channel 830 is a channel for the real-time service 832 to send IO data 847 to the computing device 114 and receive IO data 847 from the computing device 114.

[0139] According to some examples, the real-time service 832 uses protocols for communications 856, 858, and 859 including: local WiFi, such as Institute of Electrical and Electronics Engineers (IEEE) 802.11; WebRTC, which is an HTML5 specification designed for real-time media communications; TCP / UDP; Stream Control Transmission Protocol (SCTP); Quick UDP Internet Connections (QUIC), and the like.

[0140] The bandwidth required for media channel 828 can be determined as follows. For uncompressed, 4K resolution of 3840x2160 pixels, 24-bit RGB, and 30 fps, the bandwidth would require 746 MB per second. Using H264 / AVC compression at a ratio of approximately 2000:1, the bandwidth only requires 373 KB per second. Using H265 / HEVC further reduces the bandwidth to 175 KB / s, which is equivalent to 1.4 Mbps. In some embodiments, VP8 or VP9 compression is used.

[0141] The communication (comm) service 822 communicates with the communication server 902 of the computing device 114 via communication 852 and communication 854. The communication service 822 is configured to communicate with the computing device 114 via communication 852 using the USB connection 824. The communication service 822 is configured to communicate with the computing device 114 via communication 854 using the low power consumption (BLE) connection 826 communicates with computing device 114. According to some examples, communication server 822 uses other wireless protocols.

[0142] The real-time service 832 sends the AV data 846 to the codec module 820, and sends the IO data 847 to the remote desktop stream provider module 818 and receives the IO data 847. The codec module 820 is an encoder-decoder configured to perform a method for encoding and decoding the AV data 846. In some examples, the codec module 820 is configured to encode and decode according to VP8, VP9, ​​MPEG's High Efficiency Video Coding (HEVC / H.265), and / or another encoding format.

[0143] The remote desktop stream provider (RDSP) module 818 is configured to receive decoded AV data 846 from the codec module 820. The RDSP module 818 sends the decoded AV data 846 to the external texture provider module 844. The external texture provider module 844 is an engine that increases or decreases the resolution of the decoded video. The external texture provider module 844 sends the textual decoded AV data 846 to the renderer module 842 to render any graphics that need to be rendered in the textual decoded AV data 846. The AV data 846 includes 3D graphics that enable the user to see objects appearing from the virtual screen. The external texture provider module 844 adjusts the AV data 846 according to the position in the space where the virtual screen is to be displayed. For example, the size of the AV data 846 is adjusted, and adjustments are made so that the virtual screen appears to be at the appropriate angle for the user of the head wearable device 116. The external texture provider module 844 generates data that can be displayed on the display of the head wearable device at the appropriate position on the display, so that the AV data 846 appears to be in the position of the virtual screen to the user wearing the head wearable device 116. The RDSP module 818 sends the textualized decoded AV data 846 to the interpreter 817 for display. In some embodiments, the RDSP module 818 maintains a buffer of the last screen to be displayed and only receives the changed portion of the virtual screen from the AV data 846. The RDSP module 818 composes the screen data based on the buffer of screen data and the changed portion of the virtual screen from the AV data 846. In some embodiments, the RDSP module 818 decompresses the screen data from the AV data 846 sent by the computing device 114. In some embodiments, the RDSP module 818 sends the position of the head wearable device 116 to the computing device 114 so that the computing device 114 can adjust the screen data for presentation to the user.

[0144] The scanning service module 816 determines the location of the computing device 114 for reference in locating the virtual screen. The position anchor module 808 determines the location and size of the virtual screen and sends this information to the texture streaming module 810. The texture streaming module 810 causes the textual decoded AV data 846 and UI items 803 to be displayed to the user of the head wearable device 116 based on the location from the position anchor module 808.

[0145] The audio component module 812 causes the audio portion of the textually decoded AV data 846 to be played on the speakers of the head wearable device 116 and captures audio data received from the user input device 528. The captured audio data is available as input data 815 to the voice tracking module 836.

[0146] The eye tracking module 834 monitors input data 815 to track the eyes of the user of the head wearable device 116. The eye tracking module 834 sends the eye position to the input interaction controller 814. The input data 815 can come from a camera pointed at the user's eyes on the head wearable device 116. The voice tracking module 836 monitors the audio input data 815, recognizes commands, and sends the commands to the input interaction controller 814. The head tracking module 838 uses the input data 815 to track the head of the user of the head wearable device 116. The input data 815 includes data from a gyroscope. The hand tracking module 840 tracks the hands of the user of the head wearable device 116 based on data from the input devices of the head wearable device 116, including video data of the user's hands, and generates input data 815. The input data 815 indicates whether the hand tracking module 840 detects input from the user. For example, the hand tracking module 840 determines that the user's finger has touched the virtual screen. The hand tracking module 840 generates input data 815 to indicate that the user has touched the screen of the virtual screen. The input interaction controller 814 interprets the input data 815 to indicate the movement or click of the mouse on the virtual screen. The input interaction controller 814 then determines that the movement or click of the mouse on the virtual screen should be sent to the computing device 114 as IO data 847. The input interaction controller 814 sends the IO data 847 indicating the movement or click of the mouse at a specific location on the virtual screen to the RDSP module 818. The RDSP module 818 sends the IO data 847 to the real-time service 832 to be sent as IO data 847 to the computing device 114 via the control channel 830.

[0147] In some embodiments, the hand tracking functions of the hand tracking module 840 are performed by an external device. In some embodiments, the hand tracking functions of the hand tracking module 840 are performed by the core functions of the operating system of the head wearable device 116, so that the code used to implement these functions is not interpreted by the interpreter 817.

[0148] The input interaction controller 814 obtains the IO data 847 and determines whether the computing device 114 indicates that the user has expressed an intention on the computing device 114 for the head wearable device 116. The input interaction controller 814 will then cause the user's intention to be executed by the head wearable device 116.

[0149] In some embodiments, the API of the input interaction controller 814 enables developers to define the interpretation of input data 815. For example, if a user presses a key on the keyboard of the computing device 114, the developer can determine which action should be taken on the head wearable device 116.

[0150] The system UI module 802 is configured to execute user interface functions for the head wearable device 116. According to some embodiments, the input interaction controller 814 sends input data 815 to the system UI module 802 for the system UI module 802 to determine the user's intent. The user consent dialogue system module 804 executes a consent dialogue, such as asking the user whether they want to display the virtual screen.

[0151] According to some embodiments, the system UI module 802 causes user interface items to be displayed to the user of the head wearable device 116. Some of the user interface items may be for a virtual screen or associated with the computing device 114. For example, the system UI module 802 causes a recycle bin to be displayed to the user of the head wearable device 116 in addition to the virtual screen. The user's intent to place an item in the recycle bin may come from the screen of the computing device 114 or the virtual screen. Other user interface items are displayed to the user by the head wearable device 116, such as drop-down menus, system menus, etc. The system UI module 802 and / or the input interaction controller 814 determines whether the user intent is intended for the head wearable device 116 to perform, or whether the user intent (e.g., moving a file to the recycle bin) is intended for the computing device 114 to perform.

[0152] Table 1 shows example applications on the head wearable device 116. For example, an application executed by an interpreter.

[0153]

[0154]

[0155]

[0156] Table 2 shows an example API for accessing the virtual screen process of the interpreter 817. Applications will use the API to build applications for the virtual screen on the head wearable device 116.

[0157]

[0158]

[0159] Figure 9 Shown is a system 900 for monitor extension using a wearable device, according to some examples. Figure 9 Shown as Figure 1 and Figure 5 The computing device 114 is shown in FIG. The computing device 114 communicates with the Figure 8 Communicate with the head wearable device 116.

[0160] The virtual screen manager 916 creates virtual screens and integrates them with the other modules of the computing device 114. The virtual screen manager 916 indicates to the screen capture module 914 which screens, monitors, specific windows, or video transmissions are virtual screens. The screen capture module 914 captures virtual windows according to the instructions from the virtual screen manager 916 and sends the captured screens to the codec module 908. In some embodiments, the screen capture module 914 only sends the changed portion of the virtual screen. In some embodiments, the screen capture module 914 compresses the screen data. The audio capture module 910 captures audio signals from the input devices of the computing device 114. The database 912 is configured to store data such as AV data 846. According to some embodiments, the AV data 846 includes 3D graphics. 3D graphics enable applications that cannot be achieved using 2D real screens.

[0161] The codec module 820 is an encoder-decoder configured to perform methods for encoding and decoding AV data 846. In some examples, the codec module 820 is configured to encode and decode according to VP8, VP9, ​​MPEG's High Efficiency Video Coding (HEVC / H.265), and / or another encoding format. The codec module 820 encodes the AV data 846 for decoding by the codec module 820 on the head wearable device 116. The head wearable device 116 is configured to decode the AV data 846 at a rate of 60 frames per second or higher.

[0162] The communication (comm) service 902 of the computing device 114 communicates with the communication service 822 of the head wearable device 116 via communications 852 and 854. The communication service 902 is configured to communicate with the head wearable device 116 via communications 852 using a USB connection 904. The communication service 902 is configured to communicate with the head wearable device 116 via communications 854 using a low energy consumption The (BLE) connection 906 communicates with the head wearable device 116. According to some examples, the communication server 902 uses other wireless protocols.

[0163] The stream controller 918 controls the streaming of the AV data 846. For example, the stream controller 918 instructs the screen capture module 914 on how often the screen should be captured. The stream controller 918 can change the frame rate, such as 60 fps, 72 fps, 80 fps, or 90 fps, and the bit rate, such as 50 Mbps, 75 Mbps, or 100 Mbps, to maintain acceptable latency.

[0164] The desktop connection server 920 is a server that controls the transmission of AV data 846 and the transmission and reception of IO data 847. The media channel 922 is a channel for the desktop connection server 920 to transmit audio / video (AV) data 846 to the head wearable device 116 via communication 856 or communication 859. The control channel 924 is a channel for the desktop connection server 920 to transmit IO data 847 to the head wearable device 116 and to receive IO data 847 from the head wearable device 116.

[0165] The input system module 930 determines whether the user intended a command to be executed on the head wearable device 116 or on the computing device 114. The input system module 930 receives input data from the input device module 932. The input simulation module 926 obtains IO data 847 indicating the user's intention from the head wearable device 116 and causes the user's intention to be executed. For example, if the IO data 847 indicates that the user intended to resize the virtual window, then the input simulation module 926 causes the resizing of the virtual window to be executed. The input capture module 928 examines the input from the input device module 932 and determines whether any input indicates the user's intention to execute a command on the head wearable device 116. For example, a key on a keyboard may be assigned the function of dimming the display on the head wearable device 116. The input capture module 928 recognizes the user's intention and generates IO data 847 to be sent to the head wearable device 116.

[0166] The input device module 932 controls the input of the input devices of the computing device 114 and sends the input to the input capture module 928 so that the input capture module 928 can determine whether the user's intention is for the head wearable device 116. The input capture module 928 maintains a data mapping between the user's intention (e.g., pressing a key) and whether the user's intention should be sent to the head wearable device 116. The input device API 934 is an API for developers to attach user intentions to the user's input. Input devices include devices such as a mouse, keyboard, touchpad, pen, touch-sensitive screen, etc. According to some embodiments, the computing device 114 is configured to perform hand tracking and recognize hand commands.

[0167] Figure 10A user interface 1000 for configuring virtual screens is shown in accordance with some embodiments. User interface 1000 is presented to a user on display 1002 of computing device 114. Those skilled in the art will appreciate that user interface 1000 can be presented in various ways. A user has added virtual screen A 1004, virtual screen B 1006, and virtual screen C 1008 outside of the physical screen 1012 of computing device 114. The user has also added real screen 1014 within physical screen 1012. The user can move, add, and delete real screens within physical screen 1012, and can move, add, and delete virtual screens outside of real screen 1014.

[0168] Options 1010 are for configuring the virtual screen. Options 1010 include refresh rate, resolution, landscape or portrait orientation, whether the virtual screen extends the real screen 1014, and location. The virtual screen manager 916 is configured to manage the virtual screen based on the selection of the virtual screen and options as described herein.

[0169] Figure 11 A system 1100 with a virtual display is shown, according to some embodiments. A head-worn device 116 is in the form of glasses. A user wearing head-worn device 116 sees a real computing device 114 and a real display 1104. Display 1104 is an optional secondary monitor for computing device 114. Head-worn device 116 allows the user to see virtual screen A 1106, virtual screen B 1108, virtual screen C 1110, and virtual UI items 1102. Hand tracking is performed by the camera of computing device 114 and / or the camera of head-worn device 116. Head-worn device 116 allows the user to see virtual UI items 1102. The user can use virtual UI items 1102 on both computing device 114 and the virtual screens.

[0170] For example, a user may drag a file from virtual screen C 1110 or from the screen of computing device 114 to virtual UI item 1102. Hand tracking module 840 detects the movement of the hand first selecting the file and then dragging the file to virtual UI item 1102. Hand tracking module 840 sends input data 815 to input interaction controller 814 to indicate the user's intention to drag the file to virtual UI item 1102. Input interaction controller 814 determines that input data 815 is to be sent to computing device 114 and generates IO data 847 to send to computing device 114. Input event processor module 806 is configured to respond to input events, such as the pressing of a button on head wearable device 116.

[0171] The computing device 114 receives the IO data 847 via the control channel 924. The input simulation module 926 determines that the IO data 847 includes a user intent and causes the user intent to be executed on the computing device 114.

[0172] If the user expresses their user intent using an input device of the computing device 114, and the user intent is a command directed to the head wearable device 116, the input capture module 928 determines that the user intent (e.g., a key pressed to dim the display of the head wearable device 116) is directed to the head wearable device 116 and generates IO data 847 to be sent to the head wearable device 116 via communication 858 over the control channel 924. The head wearable device 116 receives the IO data 847 over the control channel 830. The remote desktop stream provider module 818 sends the IO data 847 to the input interaction controller 814, which causes the user intent to be executed on the head wearable device 116.

[0173] Figure 12 1 shows a method 1200 for using a virtual screen of a wearable device according to some embodiments. The method 1200 begins at operation 1202 by receiving an indication of a location for a virtual screen. For example, referring to Figure 8 and Figure 9 , the virtual screen manager 916 sends parameters including a positioning for displaying the virtual screen to the head wearable device 116. The parameters may indicate an offset or relative position of the virtual screen relative to the computing device 114, a fixed position within a 3D coordinate system within the real world, a relative positioning relative to the positioning of the head wearable device 116, or may indicate a relative position in another manner. The remote desktop stream provider module 818 receives the parameters. The method 1200 continues at operation 1204 by receiving screen data for the virtual screen. For example, referring to Figure 8 , the remote desktop stream provider module 818 receives AV data 846 from the computing device 114 .

[0174] Method 1200 continues at operation 1206 by determining a position in space based on the positioning included in the parameters from the virtual screen manager 916. For example, the position anchor module 808 determines the position in space based on the positioning of the head wearable device 116 and the positioning for displaying the virtual screen. Method 1200 continues at operation 1208 by causing screen data to be displayed on a display of the AR head wearable device, wherein the positioning of the screen data on the display is based on the position and the positioning of the AR head wearable device. For example, the remote desktop stream provider module 818 sends AV data 846 for the virtual screen to the external texture provider module 844 to adjust the size of the IO data 847 and / or adjust the IO data 847 to appear at an appropriate perspective based on the virtual screen positioning and the positioning of the head wearable device 116. The adjusted AV data 846 is sent to the texture streaming module 810 for display to the user of the head wearable device 116.

[0175] Method 1200 may optionally include one or more additional operations. The operations of method 1200 may be performed in a different order. One or more of the operations of method 1200 may be optional.

[0176] Figure 13 13. A method 1300 for using a virtual screen of a wearable device according to some embodiments is shown. The method 1300 begins at operation 1302 by sending an indication of a location for a virtual screen. For example, referring to Figure 9 , the virtual screen manager 916 sends the definition of the virtual screen to the head wearable device 116 via the AV data 846. The method 1300 continues at operation 1304 by sending the screen data for the virtual screen to the augmented reality (AR) head wearable device. For example, the virtual screen manager 916 causes the screen data for the virtual screen to be sent via the AV data 846.

[0177] Method 1300 may optionally include one or more additional operations. The operations of method 1300 may be performed in a different order. One or more of the operations of method 1300 may be optional.

[0178] Figure 14 is a perspective view of a head wearable device in the form of glasses 1400, according to some examples. Glasses 1400 are an article of eyewear that includes electronics that operate within a network system for transmitting image and video content. Figure 14An embodiment of a head-wearable device 116 is shown. In some examples, the wearable electronic device is referred to as AR glasses. Glasses 1400 may include a frame 1432 made of any suitable material, such as plastic or metal, including any suitable shape memory alloy. Frame 1432 may have a front piece 1433 that may include a first or left lens, a display or optical element holder 1436, and a second or right lens, a display, or optical element holder 1437 connected by a bridge 1438. Front piece 1433 further includes a left end 1441 and a right end 1442. A first or left optical element 1444 and a second or right optical element 1443 may be disposed within respective left and right optical element holders 1436, 1437. Each of optical elements 1443, 1444 may be a lens, a display, a display assembly, or a combination thereof. In some examples, for example, glasses 1400 are provided with an integrated near-eye display mechanism that enables, for example, preview images of visual media captured by camera 1469 of glasses 1400 to be displayed to the user.

[0179] Frame 1432 further includes a left arm or temple piece 1446 and a right arm or temple piece 1447, which are coupled to respective left and right ends 1441, 1442 of front piece 1433 by any suitable means, such as hinges (not shown), so as to be coupled to front piece 1433 or rigidly or fixably secured to front piece 1433, thereby forming a unitary body with front piece 1433. Each of temple piece 1446 and temple piece 1447 can include: a first portion 1451 coupled to respective ends 1441 or 1442 of front piece 1433; and any suitable second portion 1452, such as a curved or arched piece, for coupling to a user's ear. In one example, front piece 1433 can be formed from a single piece of material to have a unitary or one-piece construction. In one example, the entire frame 1432 can be formed from a single piece of material to have a unitary or one-piece construction.

[0180] Eyeglasses 1400 include a computing device, such as computer 1461, which can be of any suitable type to be carried by frame 1432 and, in one example, can be of a suitable size and shape to be at least partially disposed within one or more of temple pieces 1446, 1447. In one example, computer 1461 has a size and shape similar to the size and shape of one of temple pieces 1446, 1447 and is thus disposed nearly completely, if not completely, within the structure and confines of such temple piece 1446, 1447.

[0181] In one example, computer 1461 can be disposed in both temple piece 1446 and temple piece 1447. Computer 1461 may include one or more processors with memory, wireless communication circuitry, and a power supply. Computer 1461 includes low-power circuitry, high-speed circuitry, positioning circuitry, and a display processor. Various other examples may include these elements in different configurations or integrated together in different ways. Additional details of aspects of computer 1461 may be implemented as described with reference to the following description.

[0182] The computer 1461 further includes a battery 1462 or other suitable portable power source. In one example, the battery 1462 is disposed in one of the temple pieces 1446 or 1447. Figure 14 In the glasses 1400 shown in FIG, a battery 1462 is shown disposed in the left temple piece 1446 and electrically coupled to the remainder of the computer 1461 disposed in the right temple piece 1447 using a connector 1474. The one or more input and output devices may include a connector or port (not shown) accessible from the exterior of the frame 1432 suitable for charging the battery 1462, a wireless receiver, transmitter, or transceiver (not shown), or a combination of such devices.

[0183] The glasses 1400 include a digital camera 1469. Although two cameras 1469 are depicted, other examples contemplate the use of a single or additional (i.e., more than two) cameras 1469. For ease of description, various features related to the camera 1469 will be further described with reference only to a single camera 1469, but it will be understood that these features may apply to both cameras 1469 in appropriate examples.

[0184] In various examples, the glasses 1400 can include any number of input sensors or peripherals in addition to the camera 1469. The front piece 1433 is provided with an outward-facing, front-facing, front or outer surface 1466 that faces forward or away from the user when the glasses 1400 are mounted on the user's face; and an opposing inward-facing, rear-facing, rear or inner surface 1467 that faces the user's face when the glasses 1400 are mounted on the user's face. Such sensors can include an inward-facing video sensor or digital imaging module, such as one that can be mounted or disposed on or within the inner surface 1467 of the front piece 1433 or elsewhere on the frame 1432 to face the camera 1469 of the user; and an outward-facing video sensor or digital imaging module, such as one that can be mounted or disposed on or within the outer surface 1466 of the front piece 1433 or elsewhere on the frame 1432 to face away from the camera 1469 of the user. Such sensors, peripherals, or peripherals can additionally include biometric sensors, position sensors, accelerometers, or any other such sensors. In some examples, a projector (not shown) is used to project images onto the inner surfaces of optical elements 1443, 1444 (or lenses) to provide a mixed reality or augmented reality experience for the user of glasses 1400.

[0185] Glasses 1400 also include an example of a camera control mechanism or user input mechanism, including a camera control button mounted on frame 1432 for tactile or manual engagement by the user. The camera control button provides a dual-mode or single-action mechanism because it can be set by the user between only two states: engaged and disengaged. In this example, the camera control button is a button that defaults to the disengaged state and can be pressed by the user to set it to the engaged state. When the camera control button is released, it automatically returns to the disengaged state.

[0186] In other examples, the single-action input mechanism may instead be provided by, for example, a touch-sensitive button including a capacitive sensor mounted adjacent to a surface of frame 1432 for detecting the presence of a user's finger to set the touch-sensitive button to an engaged state when the user touches the finger to a corresponding point on outer surface 1466 of frame 1432. It will be appreciated that the camera control button and capacitive touch button described above are merely two examples of tactile input mechanisms for single-action control of camera 1469, and that other examples may employ different single-action tactile control arrangements.

[0187] Computer 1461 is configured to perform the methods described herein. In some examples, computer 1461 is coupled to one or more antennas for receiving signals from a GNSS and circuitry for processing the signals, wherein the antennas and circuitry are housed in glasses 1400. In some examples, computer 1461 is coupled to one or more wireless antennas and circuitry for sending and receiving wireless signals, wherein the antennas and circuitry are housed in glasses 1400. In some examples, multiple sets of antennas and circuitry are housed in glasses 1400. In some examples, the antennas and circuitry are configured to process signals according to, for example, Bluetooth. TM , Bluetooth Low Energy TM , IEEE 802, IEEE 802.11az / be, and other communication protocols. In some examples, the PDR sensor is housed in glasses 1400 and coupled to computer 1461. In some examples, glasses 1400 is a VR headset, where optical elements 1443 and 1444 are opaque screens for displaying images to a user of the VR headset. In some examples, computer 1461 is coupled to user interface elements such as a slider or touchpad 1476 and a button 1478. Pressing and holding button 1478 resets glasses 1400. Slider or touchpad 1476 and button 1478 are used by the user to provide input to computer 1461 and / or other electronic components of glasses 1400. Glasses 1400 include one or more microphones 1482 coupled to computer 1461. Glasses 1400 also include one or more gyroscopes 1480.

[0188] in conclusion

[0189] The virtual screen or virtual monitor reduces costs because only the head wearable device 116 is required instead of several physical monitors. In addition, since the virtual monitor does not require cables, the user has better mobility. In addition, the head wearable device 116 enables the computing device 114 to be controlled by gestures, which are recognized by the head wearable device 116 and sent to the computing device 114. The virtual screen can depict 3D objects that appear from the screen. In addition, the user interface of the computing device 114 can map user interface items to perform functions on the head wearable device 116. The virtual screen can improve posture by positioning the virtual screen to encourage the user to sit up straight. The virtual screen improves productivity by enabling users to use the appropriate number of screens for the task they are performing.

[0190] Glossary

[0191] "Carrier signal" refers to any intangible medium that can store, encode, or carry instructions for execution by a machine and includes, for example, digital or analog communication signals, or other intangible medium that facilitates the transmission of such instructions. Instructions can be sent or received over a network using a transmission medium via a network interface device.

[0192] "Client device" refers to any machine that interfaces with a communications network to obtain resources from one or more server systems or other client devices, for example. A client device may be, but is not limited to, a mobile phone, a desktop computer, a laptop computer, a portable digital assistant (PDA), a smartphone, a tablet computer, an ultrabook, a netbook, a laptop computer, a multiprocessor system, a microprocessor-based or programmable consumer electronics product, a game console, a set-top box, or any other communications device that a user can use to access a network.

[0193] "Communications network" refers to, for example, one or more parts of a network, which may be an ad hoc network, an intranet, an extranet, a virtual private network (VPN), a local area network (LAN), a wireless LAN (WLAN), a wide area network (WAN), a wireless WAN (WWAN), a metropolitan area network (MAN), the Internet, a part of the Internet, a part of the Public Switched Telephone Network (PSTN), a Plain Old Telephone Service (POTS) network, a cellular telephone network, a wireless network, The coupling may be a network, other type of network, or a combination of two or more such networks. For example, the network or a portion of the network may include a wireless network or a cellular network, and the coupling may be a code division multiple access (CDMA) connection, a global system for mobile communications (GSM) connection, or other type of cellular or wireless coupling. In this example, the coupling may implement any of various types of data transmission technologies, such as single carrier radio transmission technology (1xRTT), evolution data optimized (EVDO) technology, general packet radio service (GPRS) technology, enhanced data rates for GSM evolution (EDGE) technology, the third generation partnership project (3GPP) including 3G, fourth generation wireless (4G) networks, universal mobile telecommunications system (UMTS), high speed packet access (HSPA), world wide interoperability for microwave access (WiMAX), long term evolution (LTE) standards, other data transmission technologies defined by various standards setting organizations, other long distance protocols, or other data transmission technologies.

[0194] "Component" refers to a logic or device, a physical entity, for example, having the following boundaries, which are defined by function or subroutine calls, branch points, APIs, or other technologies that provide partitioning or modularization for specific processing or control functions. A component can be combined with other components via its interface to perform machine processing. A component can be a packaged functional hardware unit designed for use with other components, and a part of a program that generally performs a specific function of a related function. A component can constitute a software component (for example, a code embodied on a machine-readable medium) or a hardware component. A "hardware component" is a tangible unit that can perform certain operations and can be configured or arranged in a certain physical manner. In various examples, one or more computer systems (for example, a stand-alone computer system, a client computer system, or a server computer system) or one or more hardware components (for example, a processor or a processor group) of a computer system can be configured by software (for example, an application or an application part) to operate to perform certain operations as described herein. Hardware components can also be implemented mechanically, electronically, or in any suitable combination thereof. For example, a hardware component can include a dedicated circuit or logic that is permanently configured to perform certain operations. The hardware component can be a special-purpose processor, such as a field programmable gate array (FPGA) or an application-specific integrated circuit (ASIC). The hardware component can also include a programmable logic or circuit that is temporarily configured to perform certain operations by software. For example, the hardware component can include software executed by a general-purpose processor or other programmable processor. Once configured by such software, the hardware component becomes a specific machine (or a specific component of a machine), which is uniquely customized to perform the configured function and is no longer a general-purpose processor. It will be appreciated that it can be decided whether to mechanically implement the hardware component in a dedicated and permanently configured circuit or in a temporarily configured (e.g., configured by software) circuit for cost and time considerations. Therefore, the phrase "hardware component" (or "hardware-implemented component") should be understood to include tangible entities, i.e., entities that are physically constructed, permanently configured (e.g., hardwired) or temporarily configured (e.g., programmed) to operate in some way or perform certain operations described herein. Considering an example in which a hardware component is temporarily configured (e.g., programmed), it is not necessary to configure or instantiate each hardware component in the hardware component at any one time. For example, in the case where a hardware component includes a general-purpose processor that is configured by software to become a special-purpose processor, the general-purpose processor can be configured as a different special-purpose processor (e.g., including different hardware components) at different times. The software configures a specific one or more processors accordingly, such as to constitute a specific hardware component at one time and to constitute a different hardware component at a different time. A hardware component can provide information to other hardware components and receive information from other hardware components. Therefore, the described hardware components can be considered to be communicatively coupled.When there are multiple hardware components at the same time, communication can be achieved by signal transmission between or among two or more hardware components (for example, by appropriate circuits and buses). In the example where multiple hardware components are configured or instantiated at different times, communication between such hardware components can be achieved, for example, by storing information in a memory structure accessible to multiple hardware components and retrieving information in the memory structure. For example, a hardware component can perform an operation, and the output of the operation is stored in a memory device coupled to its communication ground. Then, other hardware components can access the memory device at a subsequent time to retrieve the stored output and process it. The hardware component can also initiate communication with an input device or an output device, and can operate on resources (for example, a collection of information). The various operations of the example methods described herein can be performed at least in part by temporarily configuring (for example, by software) or permanently configuring one or more processors to perform related operations. Whether it is temporarily configured or permanently configured, such a processor can constitute a processor-implemented component that operates to perform one or more operations or functions described herein. As used herein, a "processor-implemented component" refers to a hardware component implemented using one or more processors. Similarly, the method described in this article can be implemented at least in part by a processor, wherein one or more specific processors are examples of hardware. For example, at least some operations in each operation of the method can be performed by one or more processors or the parts implemented by the processor. In addition, one or more processors can also operate to support the execution of related operations in a "cloud computing" environment or operate as "software as a service" (SaaS). For example, at least some operations in the operation can be performed by a computer group (as an example of a machine including a processor), wherein these operations can be accessed via a network (for example, the Internet) and via one or more appropriate interfaces (for example, API). The execution of some operations can be distributed between processors, not only resident in a single machine, but deployed across multiple machines. In some examples, a processor or the parts implemented by a processor can be located in a single geographical location (for example, in a home environment, an office environment or a server farm). In other examples, a processor or the parts implemented by a processor can be distributed across multiple geographical locations.

[0195] "Computer-readable storage media" refers to, for example, both machine storage media and transmission media. Thus, these terms encompass both storage devices / media and carrier / modulated data signals. The terms "machine-readable medium," "computer-readable medium," and "device-readable medium" mean the same thing and may be used interchangeably in this disclosure.

[0196] A "transient message" is a message that is accessible for a limited duration, for example. Transient messages can be text, images, videos, and the like. The access period for a transient message can be set by the sender. Alternatively, the access period can be a default setting or a setting specified by the recipient. Regardless of the setting technique, the message is transient.

[0197] “Machine storage media” refers to, for example, a single or multiple storage devices and media (e.g., centralized or distributed databases, and associated caches and servers) that store executable instructions, routines, and data. Thus, the term should be taken to include, but is not limited to, solid-state memory and optical and magnetic media, including memory internal or external to the processor. Specific examples of machine storage media, computer storage media, and device storage media include: non-volatile memory, including, for example, semiconductor memory devices such as erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), FPGAs, and flash memory devices; magnetic disks, such as internal hard disks and removable disks; magneto-optical disks; and CD-ROM and DVD-ROM disks. The terms “machine storage media,” “device storage media,” and “computer storage media” mean the same thing and are used interchangeably in this disclosure. The terms “machine storage media,” “computer storage media,” and “device storage media” expressly exclude carrier waves, modulated data signals, and other such media, at least some of which are encompassed by the term “signal media.”

[0198] “Non-transitory computer-readable storage medium” refers to a tangible medium that is capable of storing, encoding, or carrying instructions for execution by a machine, for example.

[0199] "Signal medium" refers to any intangible medium that is capable of storing, encoding, or carrying instructions for execution by a machine, and includes digital or analog communication signals or other intangible media that facilitate the communication of software or data. The term "signal medium" should be taken to include any form of modulated data signal, carrier wave, etc. The term "modulated data signal" means a signal that has one or more of its characteristics set or changed in such a manner as to encode information in the signal. The terms "transmission medium" and "signal medium" mean the same thing and may be used interchangeably in this disclosure.

[0200] "User device" means, for example, a device that a user accesses, controls, or possesses and interacts with to perform actions or interact with other users or computer systems. Additional claimable subject matter includes the following:

[0201] Example 1 is an augmented reality (AR) head wearable device, comprising: a processor; and a memory storing instructions that, when executed by the processor, configure the AR head wearable device to perform operations, the operations comprising: receiving an indication of a positioning for a virtual screen; receiving screen data for the virtual screen; determining a position in space based on the positioning; and causing the screen data to be displayed on a display of the AR head wearable device, wherein the positioning of the screen data on the display is based on the position and the positioning of the AR head wearable device.

[0202] In Example 2, the subject matter of Example 1 includes receiving screen data from a computing device, and wherein the operations further include: receiving input from a user of an AR head wearable; determining that the input indicates an operation intended for the computing device; and sending the input to the computing device.

[0203] In Example 3, the subject matter of Example 2 includes, wherein the operations further comprise tracking the user's hands, eyes, voice, or head to determine input from the user.

[0204] In Example 4, the subject matter of any one of Examples 1 to 3 includes, wherein the operations further comprise: receiving input from a computing device; and performing an action indicated by the input on the AR head wearable.

[0205] In Example 5, the subject matter of Example 4 includes, wherein the indication of the positioning is an offset relative to a screen of the computing device or a relative position relative to an AR head wearable.

[0206] In Example 6, the subject matter of any one of Examples 1 to 5 includes, wherein the operations further comprise adjusting the screen data according to the position in space.

[0207] In Example 7, the subject matter of any one of Examples 1 to 6 includes, wherein the position is a first position, and wherein the operation further comprises: obtaining a second position for the user interface item, the second position being different from the position of the computing device and the first position, wherein screen data is received from the computing device; causing the user interface item to be displayed on a display of the AR head wearable device, wherein the positioning of the user interface item on the display is based on the second position and the positioning of the AR head wearable device.

[0208] In Example 8, the subject matter of any one of Examples 1 to 7 includes, wherein the operations further include: determining that the user intent is intended for the AR head wearable device; and causing the user intent to be performed on the AR head wearable device.

[0209] In Example 9, the subject matter of any one of Examples 1 to 8 includes, wherein the operations further comprise receiving an indication of a refresh rate for the virtual screen and a size of the virtual screen.

[0210] In Example 10, the subject matter of any one of Examples 1 to 9 includes wherein the screen data is received from a server computer, and the screen data includes a video.

[0211] In Example 11, the subject matter of any one of Examples 1 to 10 includes, wherein the display of the AR head wearable device includes a first image display associated with a left side of the AR head wearable device and a second image display associated with a right side of the AR head wearable device.

[0212] In Example 12, the subject matter of Example 11 includes, wherein the operation further comprises: adjusting the screen data according to the position in the space for the first image display to generate first adjusted screen data; adjusting the screen data according to the position in the space for the second image display to generate second adjusted screen data; and causing the first adjusted screen data to be displayed on the first image display, and causing the second adjusted screen data to be displayed on the second image display.

[0213] In Example 13, the subject matter of any one of Examples 1 to 12 includes, wherein the screen data is received from a computing device, and the computing device is a smartphone or a personal computing device.

[0214] In Example 14, the subject matter of any one of Examples 1 to 13 includes, wherein the screen data is received from the computing device via a computer network or via a high-speed connection between the AR head wearable and the computing device.

[0215] Example 15 is a non-transitory computer-readable storage medium comprising instructions that, when processed by an augmented reality (AR) head wearable device, configure the AR head wearable device to perform operations comprising: receiving an indication of a positioning for a virtual screen; receiving screen data for the virtual screen; determining a position in space based on the positioning; and causing the screen data to be displayed on a display of the AR head wearable device, wherein the positioning of the screen data on the display is based on the position and the positioning of the AR head wearable device.

[0216] In Example 16, the subject matter of Example 15 includes, wherein the screen data is received from a computing device, and wherein the operations further comprise: receiving input from a user of the AR head wearable; determining that the input indicates an operation intended for the computing device; and sending the input to the computing device.

[0217] Example 17 is a computing device comprising: a processor; and a memory storing instructions that, when executed by the processor, configure the computing device to perform operations comprising: sending an indication of positioning for a virtual screen; and sending screen data for the virtual screen to an augmented reality (AR) head wearable device.

[0218] In Example 18, the subject matter of Example 17 includes, wherein the operation further includes: receiving a positioning of the AR head wearable device; determining a position in space based on the positioning and the positioning of the AR head wearable device; adjusting screen data based on the position; and sending the adjusted screen data to the AR head wearable device.

[0219] In Example 19, the subject matter of any one of Examples 17 to 18 includes, wherein the operations further comprise: receiving a user intent of a user of the computing device from the AR head wearable; and causing the user intent to be executed on the computing device.

[0220] In Example 20, the subject matter of Example 19 includes, wherein the operations further comprise: determining that the user intent is intended for an AR head wearable device; and causing the user intent to be sent to the AR head wearable device.

[0221] Example 21 is at least one machine-readable medium comprising instructions that, when executed by a processing circuit, cause the processing circuit to perform operations for implementing any one of Examples 1 to 20.

[0222] Example 22 is an apparatus comprising means for implementing any one of Examples 1 to 20.

[0223] Example 23 is a system for implementing any one of Examples 1 to 20.

[0224] Example 24 is a method for implementing any one of Examples 1 to 20.

Claims

1. An augmented reality (AR) head wearable device, comprising: processor; as well as A memory storing instructions that, when executed by the processor, configure the AR head wearable device to perform operations comprising: receiving an indication of a location for the virtual screen; receiving screen data for the virtual screen; determining a position in space based on the positioning; and The screen data is caused to be displayed on a display of the AR head wearable device, wherein a positioning of the screen data on the display is based on the position and the positioning of the AR head wearable device.

2. The AR head wearable device according to claim 1, wherein: receiving the screen data from a computing device, and wherein the operations further comprise: receiving input from a user of the AR head wearable device; determining that the input indication is intended for an operation of the computing device; and The input is sent to the computing device.

3. The AR head wearable device according to claim 2, wherein: The operations further include: Tracking the user's hands, eyes, voice, or head to determine input from the user.

4. The AR head wearable device according to claim 1, wherein: The operations further include: receiving input from a computing device; and An action indicated by the input is performed on the AR head wearable device.

5. The AR head wearable device according to claim 4, wherein: The indication of positioning is an offset relative to a screen of the computing device or a relative position relative to the AR head wearable device.

6. The AR head wearable device according to any one of claims 1 to 5, wherein: The operations further include: The screen data is adjusted according to the position in the space.

7. The AR head wearable device according to any one of claims 1 to 5, wherein: The position is a first position, and wherein the operations further comprise: obtaining a second position for a user interface item, the second position being different from a position of a computing device and the first position, wherein the screen data is received from the computing device; causing the user interface item to be displayed on a display of the AR head wearable device, wherein a positioning of the user interface item on the display is based on the second position and a positioning of the AR head wearable device; determining a user intent by which the user indicates association with the user interface item; and In response to determining that the user intent is intended for the computing device, The user intent is sent to the computing device.

8. The AR head wearable device according to any one of claims 1 to 5, wherein: The operations further include: determining that the user intent is directed to the AR head wearable device; and The user intends to execute on the AR head wearable device.

9. The AR head wearable device according to any one of claims 1 to 5, wherein: The operations further include: An indication of a refresh rate for the virtual screen and a size of the virtual screen is received.

10. The AR head wearable device according to any one of claims 1 to 5, wherein: The screen data is received from a server computer, and the screen data includes a video.

11. The AR head wearable device according to any one of claims 1 to 5, wherein: The display of the AR head wearable device includes a first image display associated with a left side of the AR head wearable device and a second image display associated with a right side of the AR head wearable device.

12. The AR head wearable device according to claim 11, wherein: The operations further include: adjusting the screen data according to a position in space for the first image display to generate first adjusted screen data; adjusting the screen data according to the position in space for the second image display to generate second adjusted screen data; and The first adjusted screen data is caused to be displayed on the first image display, and the second adjusted screen data is caused to be displayed on the second image display.

13. The AR head wearable device according to any one of claims 1 to 5, wherein: The screen data is received from a computing device, and the computing device is a smart phone or a personal computing device.

14. The AR head wearable device according to any one of claims 1 to 5, wherein: The screen data is received from the computing device via a computer network or via a high-speed connection between the AR head wearable device and the computing device.

15. A non-transitory computer-readable storage medium comprising instructions that, when processed by an augmented reality (AR) head wearable device, configure the AR head wearable device to perform operations comprising: receiving an indication of a location for the virtual screen; receiving screen data for the virtual screen; determining a position in space based on the positioning; as well as The screen data is caused to be displayed on a display of the AR head wearable device, wherein a positioning of the screen data on the display is based on the position and the positioning of the AR head wearable device.

16. The non-transitory computer-readable storage medium of claim 15, wherein: receiving the screen data from a computing device, and wherein the operations further comprise: receiving input from a user of the AR head wearable device; determining that the input indication is intended for an operation of the computing device; and The input is sent to the computing device.

17. A computing device comprising: processor; as well as a memory storing instructions that, when executed by the processor, configure the computing device to perform operations comprising: sending an indication of positioning for the virtual screen; as well as Screen data for the virtual screen is sent to an augmented reality (AR) head wearable device.

18. The computing device of claim 17, wherein: The operations further include: Receiving a location of the AR head wearable device; determining a position in space based on the positioning and the positioning of the AR head wearable device; adjusting screen data based on the position; and The adjusted screen data is sent to the AR head wearable device.

19. The computing device according to claim 17 or 18, wherein: The operations further include: receiving a user intent of a user of the computing device from the AR head wearable; and The user intent is caused to execute on the computing device.

20. The computing device of claim 19, wherein: The operations further include: Determining that the user intent is intended for the AR head wearable device; and The user intention is sent to the AR head wearable device.

Images