Orientation of enhanced content in interactive system

Abstract

The head-mounted device determines an imaginary reference plane that intersects the head of a user viewing enhanced content in a viewing pane having a vertical size and a lateral size in a display of the head-mounted device. The imaginary reference plane coincides with a first viewing direction of the user's head. The apparatus detects rotational movement of a user's head in a vertical direction while viewing enhanced content. In response to the detected rotational movement, when the augmented content is viewed in a second viewing direction, the apparatus determines the second viewing direction of the user's head and determines a reference angle between the imaginary reference plane and the second viewing direction. Based on the reference angle, the device assigns one (or a combination) of a display board display mode and a head lock display mode to enhanced content presented in the display.

Description

[0001] CLAIM OF PRIORITY

[0002] This application claims the benefit of priority to U.S. Patent Application Serial No. 18 / 187,594, filed March 21, 2023, which is incorporated by reference herein in its entirety. BACKGROUND

[0003] The present disclosure relates generally to wearable devices, including headsets for viewing augmented content displayed in a content interaction system. The present disclosure also relates to orienting content in an augmented reality (AR) display, for example, in a heads-up display or a headset. BRIEF DESCRIPTION OF DRAWINGS

[0004] In the drawings, which are not necessarily drawn to scale, like numerals can describe similar components in different views. To easily identify the discussion of any particular element or action, one or more of the highest three digits in a figure number are often used to unlock the figure in which that element was first introduced. Some non-limiting examples are shown in the drawings, in which:

[0005] Figure 1 is a diagrammatic representation of a networked environment in which the present disclosure can be deployed according to some examples.

[0006] Figure 2 is a diagrammatic representation of an interaction system having both client-side and server-side functionality according to some examples.

[0007] Figure 3 is a diagrammatic representation of a data structure as maintained in a database according to some examples.

[0008] Figure 4 is a diagrammatic representation of a message according to some examples.

[0009] Figure 5 components of a headset according to some examples are shown.

[0010] Figure 6 aspects of a display card display mode according to some examples are shown.

[0011] Figure 7 aspects of a headlock display mode according to some examples are shown.

[0012] Figure 8A an example view in a display card display mode is shown.

[0013] Figure 8B another example view in a display card display mode is shown.

[0014] Figure 8CAnother example view is shown in a display card display mode.

[0015] Figure 9 An example view is shown in a head lock display mode.

[0016] Figure 10 Aspects of a hybrid display mode are shown in accordance with some examples.

[0017] Figure 11 A method 1100 of orienting content in an AR display is shown in accordance with one embodiment.

[0018] Figure 12 is a diagrammatic representation of a machine in the form of a computer system within which a set of instructions can be executed to cause the machine to perform any one or more of the methods discussed herein.

[0019] Figure 13 is a block diagram illustrating a software architecture, wherein an example can be implemented. DETAILED DESCRIPTION

[0020] Users with various interests from various locations can capture digital images of various subjects and make the captured images available to others via a network, such as the Internet. To enhance the user's experience with digital images and provide various features, it can be challenging and computationally intensive for a computing device to enable performing image processing operations on various objects and / or features captured under various varying conditions (e.g., variations in image scale, noise, illumination, motion, or geometric distortion).

[0021] AR technology aims to bridge the gap between virtual and real-world environments by providing an augmented real-world environment that is enhanced with electronic information. Thus, the electronic information appears to be part of the real-world environment as perceived by the user. In examples, AR technology also provides a user interface to interact with the electronic information overlaid in the augmented real-world environment.

[0022] As mentioned above, the present disclosure also relates to orienting content in an AR display, such as in a heads-up display or a head-mounted device. A piece of content that is oriented in front of and towards the user in an AR display can be referred to as head-locked content. However, content positioned in this way can feel like it is attached to the display screen rather than the real world, thereby reducing the sense of realism or immersive user experience.

[0023] A common solution to try to increase realism is to rotate content laterally around the user so that as the user's head rotates, the content consistently faces the user's head. However, as the user looks up or down, content aligned in this way can become increasingly invisible. In extreme cases, if the user significantly raises or lowers his or her head while seeking to view AR content, the content in the augmented display can reduce or fold into a single edge.

[0024] AR systems enable real and virtual environments to be combined to varying degrees to facilitate interaction from a user in real-time. As described herein, such AR systems can thus include various possible combinations of real and virtual environments and be closer to a real environment than a virtual environment (e.g., without real elements) including AR that primarily includes real elements. In this way, a real environment can be connected with a virtual environment through an AR system. A user immersed in an AR environment can navigate through such an environment, and the AR system can track the user's viewpoint to provide visualizations based on the user's location in the environment. As described by implementations herein, AR experiences can be provided in an interactive system.

[0025] Implementations of the technology described herein enable various operations involving AR content for capturing, mirroring, modifying, and navigating such content with a given electronic device (e.g., a mobile computing device) or head-mounted apparatus (e.g., glasses). Some implementations herein generally relate to wearable devices including head-mounted apparatuses that include waveguide-based displays. The disclosure also relates to mirroring AR content in wearable devices and mirroring and navigating AR content in an interactive system.

[0026] Interactive systems, such as messaging systems, are frequently used and increasingly used by users of mobile computing devices in various environments to provide different types of functionality in a convenient manner. As described herein, interactive systems include practical applications that provide improvements in capturing and mirroring image data and rendering AR content (e.g., images, videos, etc.) based on captured image data by providing technical improvements in capturing image data at least with electronic devices that are limited in power and resources. Such improvements in capturing and mirroring image data are achieved by the technology provided by the present technology, which reduces latency and improves efficiency in processing captured image data, thereby also reducing power consumption of the capturing device.

[0027] As further discussed herein, the underlying infrastructure supports the creation and sharing of interactive media, referred to herein as messages, including three-dimensional (3D) content or AR effects, throughout various components of the interactive system. In example implementations described herein, messages can enter the system from live cameras or via storage (e.g., where messages including 3D content and / or AR effects are stored in memory or a database). The present system supports motion sensor input as well as the loading of external effects and asset data.

[0028] As referred to herein, the phrases“AR experience,”“AR content item,”“AR content generator” include or refer to various image processing operations corresponding to image modification, filtering, AR content generators, media overlays, transformations, and the like as further described herein, and can also include playback of audio or music content during presentation of AR content or media content.

[0029] Networked computing environment

[0030] Figure 1 is a block diagram illustrating an example interactive system 100 for facilitating interaction (e.g., exchanging text messages, making text, audio, and video calls, or playing games) over a network. The interactive system 100 includes a plurality of client systems 102, each of which hosts a plurality of applications including an interactive client 104 and other applications 106. Each interactive client 104 is communicatively coupled to other instances of the interactive client 104 (e.g., hosted on respective other user systems 102), an interactive server system 110, and third-party servers 112 via one or more communication networks including a network 108 (e.g., the Internet). The interactive client 104 can also communicate with locally-hosted applications 106 using an application program interface (API).

[0031] Each user system 102 can include a plurality of user devices, such as a mobile device 114, a head-mounted device 116, and a computer client device 118, which are communicably connected to exchange data and messages.

[0032] The interactive client 104 interacts with other interactive clients 104 and with the interactive server system 110 via the network 108. Data exchanged between the interactive clients 104 (e.g., interactions 120) and between the interactive client 104 and the interactive server system 110 includes functions (e.g., commands to activate functions) and payload data (e.g., text, audio, video, or other multimedia data).

[0033] The interaction server system 110 provides server-side functionality to the interaction clients 104 via the network 108. While certain functionality of the interaction system 100 is described herein as being performed by the interaction clients 104 or by the interaction server system 110, the location of certain functionality, either within the interaction clients 104 or within the interaction server system 110, can be a design choice. For example, it can be technically preferable to initially deploy a particular technology or functionality within the interaction server system 110, but later migrate that technology or functionality to the interaction clients 104 where the user systems 102 have sufficient processing power.

[0034] The interaction server system 110 supports various services and operations that are provided to the interaction clients 104. Such operations include sending data to the interaction clients 104, receiving data from the interaction clients 104, and processing data generated by the interaction clients 104. The data can include message content, client device information, geolocation information, media enhancements and overlays, message content persistence conditions, social network information, and live event information. Data exchange within the interaction system 100 is activated by and controlled through functionality available via the user interface of the interaction clients 104.

[0035] Turning now specifically to the interaction server system 110, an API server 122 is coupled to, and provides a programmatic interface to, an interaction server 124 that makes functions of the interaction server 124 available to the interaction clients 104, other applications 106, and third-party servers 112. The interaction server 124 is communicatively coupled to a database server 126 that facilitates access to a database 128 that stores data associated with interactions processed by the interaction server 124. Similarly, a web server 130 is coupled to the interaction server 124, and provides an web-based interface to the interaction server 124. In this regard, the web server 130 processes incoming network requests over the Hypertext Transfer Protocol (HTTP) and several other related protocols.

[0036] The API server 122 receives and sends interaction data (e.g., command and message payloads) between the interaction server 124 and the client system 102 (and, e.g., the interaction client 104 and other applications 106) and the third-party servers 112. Specifically, the API server 122 provides a set of interfaces (e.g., routines and protocols) that the interaction client 104 and other applications 106 can call or query to activate functionality of the interaction server 124. The API server 122 exposes various functions supported by the interaction server 124, including account registration; login functionality; sending interaction data from a particular interaction client 104 to another interaction client 104 via the interaction server 124; transferring media files (e.g., images or videos) from the interaction client 104 to the interaction server 124; setting a collection of media data (e.g., a story); retrieving a list of friends of a user of the user system 102; retrieving messages and content; adding and deleting entities (e.g., friends) to an entity graph (e.g., a social graph); locating friends in a social graph; and opening an application event (e.g., related to the interaction client 104).

[0037] The interaction server 124 hosts a number of systems and subsystems, described below with reference to Figure 2

[0038] Linked applications

[0039] Returning to the interaction client 104, features and functionality of external resources (e.g., linked applications 106 or widgets) are made available to users via the interface of the interaction client 104. In this context, “external” refers to the fact that the application 106 or widget is external to the interaction client 104. External resources are typically provided by third parties, but can also be provided by the creator or provider of the interaction client 104. The interaction client 104 receives user selections of options to launch or access features of such external resources. The external resource can be an application 106 installed on the user system 102 (e.g., a “native app”), or a scaled-down version of an application (e.g., a “widget”) hosted on the user system 102 or located remotely from the user system 102 (e.g., on a third-party server 112). The scaled-down version of an application includes a subset of the features and functionality of the application (e.g., of a full-scale native version of the application), and is implemented using markup language documents. In some examples, the scaled-down version of an application (e.g., a “widget”) is a web-based markup language version of the application, and is embedded in the interaction client 104. In addition to using markup language documents (e.g.,.html files), a widget can include script language (e.g.,.js files or.json files) and style sheets (e.g.,.ss files).

[0040] ​In response to receiving a user selection of an option to launch or access a feature for an external resource, the interaction client 104 determines whether the selected external resource is a web-based external resource or a locally installed application 106. In some cases, an application 106 that is locally installed on the user system 102 can be launched independently of and separately from the interaction client 104, such as by selecting an icon corresponding to the application 106 on a home screen of the user system 102. A small-scale version of such an application can be launched or accessed via the interaction client 104, and in some examples, no part of the small-scale application can be accessed outside of the interaction client 104 or only limited portions of the small-scale application can be accessed outside of the interaction client 104. The small-scale application can be launched by the interaction client 104 receiving a markup language document associated with the small-scale application from the third-party server 112 and processing such a document.

[0041] In response to determining that the external resource is a locally installed application 106, the interaction client 104 instructs the user system 102 to launch the external resource by executing locally stored code corresponding to the external resource. In response to determining that the external resource is a web-based resource, the interaction client 104 communicates with the third-party server 112, for example, to obtain a markup language document corresponding to the selected external resource. The interaction client 104 then processes the obtained markup language document to present the web-based external resource within the user interface of the interaction client 104.

[0042] The interaction client 104 can notify a user of the user system 102 or other users (e.g., "friends") associated with such a user of activity occurring in one or more external resources. For example, the interaction client 104 can provide participants in a conversation (e.g., a chat session) in the interaction client 104 with notifications related to one or more members of a group of users currently or recently using an external resource. One or more users can be invited to join an active external resource or launch a recently used but not currently active external resource (in a group of friends). An external resource can provide participants in a conversation each using a respective interaction client 104 with the ability to share an item, condition, state, or location in the external resource with one or more members of a group of users in a chat session. The shared item can be an interactive chat card that members of the chat can utilize to interact, such as to launch the corresponding external resource, view particular information within the external resource, or bring the members of the chat to a particular location or state within the external resource. Within a given external resource, a response message can be sent to a user on the interaction client 104. Based on the current context of the external resource, the external resource can selectively include different media items in the response.

[0043] The interaction client 104 can present a list of available external resources (e.g., applications 106 or widgets) to the user to launch or access a given external resource. The list can be presented in the manner of a contextually relevant menu. For example, icons representing different applications (or widgets) of the applications 106 (or widgets) can vary based on how the user launches the menu (e.g., from a conversational interface or from a non-conversational interface).

[0044] System Architecture

[0045] Figure 2 is a block diagram illustrating additional details regarding the interaction system 100 according to some examples. In particular, the interaction system 100 is shown to include the interaction client 104 and the interaction server 124. The interaction system 100 includes a number of subsystems that are supported on the client side by the interaction client 104 and on the server side by the interaction server 124. Example subsystems are discussed below.

[0046] The image processing system 202 provides various functionality that enables a user to capture and enhance (e.g., annotate or otherwise modify or edit) media content associated with a message.

[0047] The camera system 204 includes control software (e.g., in a camera application) that interacts with and controls the camera hardware of the user system 102 (e.g., directly or via operating system controls) to modify and enhance live images captured and displayed via the interaction client 104.

[0048] The augmentation system 206 provides functionality related to the generation and publication of augmentations (e.g., media overlays) for images captured in real-time by the camera of the user system 102 or images retrieved from the memory of the user system 102. For example, the augmentation system 206 is operable to select, present, and display media overlays (e.g., image filters or image lenses) for the interaction client 104 for use in augmenting live images received via the camera system 204 or stored images retrieved from the memory of the user system 102. These augmentations are selected by the augmentation system 206 based on some inputs and data, for example:

[0049] • the geographic location of the user system 102; and

[0050] • social network information of the user of the user system 102.

[0051] The augmentations can include audio and visual content as well as visual effects. Examples of audio and visual content include pictures, text, logos, animations, and sound effects. Examples of visual effects include color overlays. The audio and visual content or visual effects can be applied to media content items (e.g., photos or videos) at the user system 102 for communication in a message or to video content, such as a video content stream or feed sent from the interactive client 104. Thus, the image processing system 202 can interact with and support various subsystems of the communication system 208, such as the messaging system 210 and the video communication system 212.

[0052] The media overlays can include text or image data that can be overlaid on a photo taken by the user system 102 or a video stream made by the user system 102. In some examples, the media overlays can be location overlays (e.g., Venice Beach), name of a live event, or a business name overlay (e.g., Beachside Café). In further examples, the image processing system 202 uses the geographic location of the user system 102 to identify a media overlay that includes the name of a business at the geographic location of the user system 102. The media overlays can include other indicia associated with the business. The media overlays can be stored in the database 128 and accessed by the database server 126.

[0053] The image processing system 202 provides a user-based publishing platform that enables users to select a geographic location on a map and upload content associated with the selected geographic location. The user can also specify circumstances under which a particular media overlay should be provided to other users. The image processing system 202 generates a media overlay that includes the uploaded content and associates the uploaded content with the selected geographic location.

[0054] The augmentation creation system 214 supports an AR developer platform and includes an application for content creators (e.g., artists and developers) to create and publish augmentations (e.g., AR experiences) for the interactive client 104. The augmentation creation system 214 provides a library of built-in features and tools for content creators, including, for example, custom shaders, tracking techniques, and templates.

[0055] In some examples, the augmentation creation system 214 provides a business-based publishing platform that enables businesses to select particular augmentations associated with geographic locations via a bidding process. For example, the augmentation creation system 214 associates the media overlays of the highest bidding businesses with corresponding geographic locations for a predefined amount of time.

[0056] The content targeting system 234 is operationally responsible (in some examples, in conjunction with the image processing system 202) for targeting content viewed in an augmented content display, for example, in the head-mounted device 116 described further below. The content targeting system 234 can be located in the interaction system 100, on the mobile device 114, in the head-mounted device 116, or a combination of these locations. The content targeting system 234 is used to provide the mixed display modes discussed herein, including, for example, the display board display mode and the headlock display mode.

[0057] The communication system 208 is responsible for enabling and handling various forms of communication and interaction within the interaction system 100, and includes a messaging system 210, an audio communication system 216, and a video communication system 212. The messaging system 210 is responsible for enforcing temporary or time-limited access to content by the interaction clients 104. The messaging system 210 includes a plurality of timers (e.g., in a transient timer system 218) that selectively enable access (e.g., for presentation and display) of messages and associated content via the interaction clients 104 based on a duration and display parameters associated with a message or collection of messages (e.g., a story). Additional details regarding the operation of the transient timer system 218 are provided below. The audio communication system 216 enables and supports audio communication (e.g., real-time audio chat) between multiple interaction clients 104. Similarly, the video communication system 212 enables and supports video communication (e.g., real-time video chat) between multiple interaction clients 104.

[0058] The user management system 220 is operationally responsible for managing user data and profiles, and includes a social network system 222 that maintains information about relationships between users of the interaction system 100.

[0059] The collection management system 224 is operationally responsible for managing collections or sets of media (e.g., sets of text, image, video, and audio data). Collections of content (e.g., messages, including images, videos, text, and audio) can be organized into "event libraries" or "event stories." Such collections can be made available for a specified period of time (e.g., the duration of an event to which the content relates). For example, content related to a concert can be made available as a "story" for the duration of the concert. The collection management system 224 can also be responsible for publishing icons to the user interface of the interactive client 104 that provide notifications of particular collections. The collection management system 224 includes curation functionality that enables a curator to manage and curate particular collections of content. For example, a curation interface enables an event organizer to curate a collection of content related to a particular event (e.g., to delete inappropriate content or redundant messages). In addition, the collection management system 224 employs machine vision (or image recognition technology) and content rules to automatically curate collections of content. In certain examples, users can be paid compensation for including user-generated content into a collection. In such cases, the collection management system 224 operates to automatically pay such users for use of their content.

[0060] The map system 226 provides various geographic location functionality and supports the presentation of map-based media content and messages by the interactive client 104. For example, the map system 226 enables the display of user icons or avatars (e.g., stored in the profile data 302) on a map to indicate the current or past locations of a user's "friends" within the context of a map, as well as media content (e.g., collections of messages including photos and videos) generated by such friends. For example, on a map interface of the interactive client 104, messages posted by a user from a particular geographic location to the interactive system 100 can be displayed to the user's "friends" within the context of that particular location on the map. A user can also share his or her location and status information with other users of the interactive system 100 via the interactive client 104 (e.g., using an appropriate status avatar), where the location and status information is similarly displayed to selected users within the context of a map interface of the interactive client 104.

[0061] The game system 228 provides various game functionality within the context of the interaction client 104. The interaction client 104 provides a game interface that provides a list of available games that can be launched by a user within the context of the interaction client 104 and played with other users of the interaction system 100. The interaction system 100 also enables a particular user to invite such other users to participate in playing a particular game by issuing invitations from the interaction client 104 to the other users. The interaction client 104 also supports voice, video, and text messaging (e.g., chat) within the context of playing games, provides leaderboards for games, and also supports providing in-game rewards (e.g., game currency and items).

[0062] The external resource system 230 provides the interaction client 104 with an interface for communicating with remote servers (e.g., third-party servers 112) to launch or access external resources (i.e., applications or applets). Each third-party server 112 hosts, for example, an application or a scaled-down version of an application (e.g., a game application, a utility application, a payment application, or a ride-sharing application) based on a markup language (e.g., HTML5). The interaction client 104 can launch a web-based resource (e.g., an application) by accessing an HTML5 file from the third-party server 112 associated with the web-based resource. The application hosted by the third-party server 112 is programmed in JavaScript with a software development kit (SDK) provided by the interaction server 124. The SDK includes APIs with functionality that can be called or activated by the web-based application. The interaction server 124 hosts a JavaScript library that provides given external resource access to certain user data of the interaction client 104. HTML5 is an example of a technology for programming games, but applications and resources programmed based on other technologies can be used.

[0063] To integrate the functionality of the SDK into the web-based resource, the SDK is downloaded by the third-party server 112 from the interaction server 124 or otherwise received by the third-party server 112. Once downloaded or received, the SDK is included as part of the application code of the web-based external resource. The code of the web-based resource can then call or activate certain functionality of the SDK to integrate features of the interaction client 104 into the web-based resource.

[0064] The SDK stored on the interaction server system 110 effectively provides a bridge between external resources (e.g., applications 106 or widgets) and the interaction client 104. This gives users a seamless experience of communicating with other users on the interaction client 104 while also preserving the look and feel of the interaction client 104. To bridge the communication between external resources and the interaction client 104, the SDK facilitates communication between the third-party servers 112 and the interaction client 104. The WebView JavaScript Bridge running on the user system 102 establishes two one-way communication channels between the external resources and the interaction client 104. Messages are sent asynchronously between the external resources and the interaction client 104 via these communication channels. Each SDK function activation is sent as a message and a callback. Each SDK function is implemented by constructing a unique callback identifier and sending a message with the callback identifier.

[0065] By using the SDK, not all information from the interaction client 104 is shared with the third-party servers 112. The SDK limits which information is shared based on the needs of the external resources. Each third-party server 112 provides the interaction server 124 with an HTML5 file corresponding to the web-based external resource. The interaction server 124 can add a visual representation (e.g., a box design or other graphic) of the web-based external resource in the interaction client 104. Once the user selects the visual representation or indicates through the graphical user interface (GUI) of the interaction client 104 that the interaction client 104 is to access a feature of the web-based external resource, the interaction client 104 obtains the HTML5 file and instantiates the resource for accessing the feature of the web-based external resource.

[0066] The interaction client 104 presents a GUI (e.g., a landing page or title screen) for the external resource. During, before, or after presenting the landing page or title screen, the interaction client 104 determines whether the launched external resource has previously been authorized to access user data of the interaction client 104. In response to determining that the launched external resource has previously been authorized to access user data of the interaction client 104, the interaction client 104 presents another GUI of the external resource that includes functionality and features of the external resource. In response to determining that the launched external resource has not previously been authorized to access user data of the interaction client 104, after displaying the landing page or title screen of the external resource for a threshold period of time (e.g., 3 seconds), the interaction client 104 slides up a menu (e.g., animates the menu to emerge from the bottom of the screen to the middle or other portion of the screen) for authorizing the external resource to access user data. The menu identifies types of user data that the external resource will be authorized to use. In response to receiving a user selection of an accept option, the interaction client 104 adds the external resource to a list of authorized external resources and allows the external resource to access user data from the interaction client 104. The external resource is authorized by the interaction client 104 to access user data under an OAuth 2 framework.

[0067] The interaction client 104 controls the types of user data shared with external resources based on the types of external resources that are authorized. For example, external resources that include full-scale applications (e.g., the application 106) are provided access to a first type of user data (e.g., two-dimensional (2D) avatars of users with or without different avatar characteristics). As another example, external resources that include small-scale versions of applications (e.g., web-based versions of the application) are provided access to a second type of user data (e.g., payment information, 2D avatars of users, 3D avatars of users, and avatars with various avatar characteristics). Avatar characteristics include different ways of customizing the appearance of an avatar (e.g., different poses, facial features, clothing, etc.).

[0068] The advertising system 232 is operable to enable third parties to purchase advertisements to be presented to end users via the interaction client 104 and also handles the delivery and presentation of these advertisements.

[0069] Data Architecture

[0070] Figure 3 FIG. 3 is a diagram illustrating a data structure 300 that can be stored in the database 304 of the interaction server system 110, in accordance with certain examples. While the contents of the database 304 are illustrated as including multiple tables, it should be understood that data can be stored in other types of data structures (e.g., object-oriented databases).

[0071] The database 304 includes message data stored within a message table 306. For any particular message, the message data includes at least message sender data, message recipient (or receiver) data, and a payload. Reference is made below to Figure 3 Further details are described regarding information that can be included in messages and included within message data stored in the message table 306.

[0072] An entity table 308 stores entity data and is linked (e.g., by reference) to an entity graph 310 and the profile data 302. Entities for which records are maintained within the entity table 308 can include individuals, corporate entities, organizations, objects, locations, events, and the like. Regardless of entity type, any entity for which the interaction server system 110 stores data can be an identified entity. Each entity is provided with a unique identifier as well as an entity type identifier (not shown).

[0073] The entity graph 310 stores information about relationships and associations between entities. Such relationships can be social, professional (e.g., working at a common company or organization), interest-based, or activity-based, merely as examples. Certain relationships between entities can be one-way, such as a subscription by a personal user to digital content of a business or publishing user (e.g., a newspaper or other digital media channel or brand). Other relationships can be two-way, such as a "friend" relationship between individual users of the interaction system 100.

[0074] Certain permissions and relationships can be attached to each relationship, and can also be attached to each direction of a relationship. For example, a two-way relationship (e.g., a friend relationship between personal users) can include authorization for publication of digital content items between the personal users, but can impose certain restrictions or filters (e.g., based on content characteristics, location data, or time-of-day data) on publication of such digital content items. Similarly, a subscription relationship between a personal user and a business user can impose varying degrees of restriction on publication of digital content from the business user to the personal user, and can significantly restrict or prevent publication of digital content from the personal user to the business user. As an example of an entity, a particular user can record certain restrictions in the record for that entity within the entity table 308 (e.g., through privacy settings). Such privacy settings can apply to all types of relationships in the context of the interaction system 100, or can selectively apply to certain types of relationships.

[0075] The profile data 302 stores various types of profile data about a particular entity. Based on privacy settings specified by the particular entity, the profile data 302 can be selectively used and presented to other users of the interaction system 100. In the case where the entity is a person, the profile data 302 includes, for example, the user's name, phone number, address, settings (e.g., notification and privacy settings), and a user-selected avatar representation (or a collection of such avatar representations). The particular user can then selectively include one or more of these avatar representations within the content of messages communicated via the interaction system 100 and on the map interface displayed by the interaction client 104 to other users. The collection of avatar representations can include a "status avatar" that presents a graphical representation of a status or activity that the user can select to communicate at a particular time.

[0076] In the case where the entity is a group, the profile data 302 for the group can similarly include one or more avatar representations associated with the group in addition to the group name, members, and various settings (e.g., notifications) for the relevant group.

[0077] The database 304 also stores augmentation data, such as overlays or filters, in an augmentation table 312. The augmentation data is associated with and applied to videos (data for which is stored in a video table 314) and images (data for which is stored in an image table 316).

[0078] In some examples, a filter is an overlay that is displayed as an overlay on an image or video during presentation to a recipient user. The filter can be of various types, including a user-selected filter from a set of filters presented by the interaction client 104 to a sending user when the sending user is composing a message. Other types of filters include a geo-location filter (also referred to as a geo-filter) that can be presented to the sending user based on a geo-location. For example, a geo-location filter specific to a nearby or special location can be presented by the interaction client 104 within a user interface based on geo-location information determined by a global positioning system (GPS) unit of the user system 102.

[0079] Another type of filter is a data filter that can be selectively presented to the sending user by the interaction client 104 based on other inputs or information gathered by the user system 102 during the message creation process. Examples of data filters include a current temperature at a particular location, a current speed at which the sending user is traveling, a battery life of the user system 102, or a current time.

[0080] Other augmentation data that can be stored within the image table 316 includes AR content items (e.g., corresponding to an applied "lens" or AR experience). The AR content items can be real-time special effects and sounds that can be added to an image or video.

[0081] Story table 318 stores data regarding collections of messages and associated image, video, or audio data that are compiled into collections (e.g., stories or galleries). The creation of a particular collection can be initiated by a particular user (e.g., each user for which a record is maintained in entity table 308). A user can create a "personal story" in the form of a collection of content that has been created and sent / broadcast by that user. To this end, the user interface of interactive client 104 can include a user-selectable icon to enable a sending user to add particular content to his or her personal story.

[0082] Collections can also constitute "live stories," which are collections of content from multiple users created manually, automatically, or using a combination of manual and automatic techniques. For example, a "live story" can constitute a curated stream of user-submitted content from various locations and events. Users whose client devices have location services enabled and are at a common location event at a particular time can be presented with an option to contribute content to a particular live story, e.g., via the user interface of interactive client 104. A live story can be identified to a user by interactive client 104 based on the user's location. The end result is a "live story" told from a group perspective.

[0083] Another type of collection of content is referred to as a "location story," which enables users whose user systems 102 are located within a particular geographic location (e.g., on a college or university campus) to contribute to a particular collection. In some examples, contributions to a location story can employ secondary authentication to verify that an end user belongs to a particular organization or other entity (e.g., is a student in a university campus).

[0084] As mentioned above, video table 314 stores video data that, in some examples, is associated with messages for which records are maintained within message table 306. Similarly, image table 316 stores image data that is associated with messages whose message data is stored in entity table 308. Entity table 308 can associate various augmentations from augmentation table 312 with various images and videos stored in image table 316 and video table 314.

[0085] Database 304 also includes [***]

[0086] Data Communication Architecture

[0087] Figure 4is a schematic diagram showing the structure of a message 400 generated by an interaction client 104 to be transmitted to a further interaction client 104 via an interaction server 124, in accordance with some examples. The content of a particular message 400 is used to populate a message table 306 stored within a database 304 accessible by the interaction server 124. Similarly, the content of a message 400 is stored in memory as “in-transit” or “in-flight” data for the user system 102 or the interaction server 124. The message 400 is shown to include the following example components:

[0088] • message identifier 402: a unique identifier that identifies the message 400.

[0089] • message text payload 404: text to be generated by a user via a user interface of the user system 102 and included in the message 400.

[0090] • message image payload 406: image data captured by a camera component of the user system 102 or retrieved from a memory component in the user system 102 and included in the message 400. Image data for a sent or received message 400 can be stored in the image table 316.

[0091] • message video payload 408: video data captured by a camera component or retrieved from a memory component of the user system 102 and included in the message 400. Video data for a sent or received message 400 can be stored in the image table 316.

[0092] • message audio payload 410: audio data captured by a microphone or retrieved from a memory component of the user system 102 and included in the message 400.

[0093] • message augmentation data 412: augmentation data (e.g., filters, stickers, or other annotations or augmentations) representing an augmentation to be applied to the message image payload 406, message video payload 408, or message audio payload 410 of the message 400. Augmentation data for a sent or received message 400 can be stored in the augmentation table 312.

[0094] • message duration parameter 414: a parameter value indicating, in seconds, an amount of time for the content (e.g., message image payload 406, message video payload 408, message audio payload 410) of the message to be presented to or made accessible to the user via the interaction client 104.

[0095] • message geolocation parameters 416: geolocation data (e.g., latitude and longitude coordinates) associated with the content payload of the message. Multiple message geolocation parameter 416 values can be included in the payload, each of which is associated with a content item included in the content (e.g., a particular image within the message image payload 406 or a particular video in the message video payload 408).

[0096] • message story identifiers 418: identifier values that identify one or more content collections (e.g.,“stories” identified in the story table 318) that are associated with particular content items in the message image payload 406 of the message 400. For example, multiple images within the message image payload 406 can each be associated with multiple content collections using identifier values.

[0097] • message tags 420: each message 400 can be tagged with multiple tags, each of which indicates a subject matter of content included in the message payload. For example, where a particular image included in the message image payload 406 depicts an animal (e.g., a lion), a tag value can be included within the message tags 420 that indicates the relevant animal. Tag values can be generated manually based on user input, or can be generated automatically using, for example, image recognition.

[0098] • message sender identifier 422: an identifier (e.g., a messaging system identifier, email address, or device identifier) that indicates a user of the user system 102 on which the message 400 was generated and from which the message 400 was sent.

[0099] • message recipient identifier 424: an identifier (e.g., a messaging system identifier, email address, or device identifier) that indicates a user of the user system 102 to which the message 400 is addressed.

[0100] The content (e.g., values) of the various components of the message 400 can be pointers to locations in tables where the content data values are stored. For example, the image values in the message image payload 406 can be pointers (or addresses) to locations within the image table 316. Similarly, the values within the message video payload 408 can point to data stored within the image table 316, the values stored within the message augmentation data 412 can point to data stored in the augmentation table 312, the values stored within the message story identifiers 418 can point to data stored in the story table 318, and the values stored within the message sender identifier 422 and the message recipient identifier 424 can point to user records stored within the entity table 308.

[0101] System with head-mounted device

[0102] Figure 5 A system 500 is shown in accordance with some examples, including a head mounted device 116 having a selector input device. Figure 5 is a high level functional block diagram of an example head mounted device 116 communicatively coupled to a mobile device 114 and various server systems 504 (e.g., an interaction server system 110) via various networks 516.

[0103] The head mounted device 116 includes one or more cameras, each of which can be, for example, a visible light camera 506, an infrared emitter 508, and an infrared camera 510.

[0104] The mobile device 114 connects with the head mounted device 116 using both a low power wireless connection 512 and a high speed wireless connection 514. The mobile device 114 also connects to the server systems 504 and the networks 516.

[0105] The head mounted device 116 also includes a two optical assembly image display 518. The two optical assembly image display 518 includes one image display associated with the left lateral side of the head mounted device 116 and one image display associated with the right lateral side of the head mounted device 116. The head mounted device 116 also includes an image display driver 520, an image processor 522, a low power circuit 524, and a high speed circuit 526. The image display 518 is used to present images and video to a user of the head mounted device 116, including images that can include a graphical user interface.

[0106] The image processor 522 includes a content orientation system 234. The content orientation system 234 is operationally responsible for orienting content viewed in an augmented content display, for example in the image display 518 of the head mounted device 116. The content orientation system 234 is used to provide the mixed display modes discussed herein, including, for example, a display card display mode and a head lock display mode.

[0107] The image display driver 520 commands and controls the image display 518. The image display driver 520 can either deliver image data directly to the image display 518 for presentation or can convert the image data into a signal or data format suitable for delivery to the image display device. For example, the image data can be video data formatted according to a compression format such as H.264 (MPEG-4 Part 10), HEVC, Theora, Dirac, RealVideo RV40, VP8, VP9, etc., while static image data can be formatted according to a compression format such as Portable Network Group (PNG), Joint Photographic Experts Group (JPEG), Tagged Image File Format (TIFF), or Exchangeable image file format (EXIF), etc.

[0108] The head-mounted device 116 includes a frame and stems (or temples) extending laterally from the frame. The head-mounted device 116 also includes a user input device 528 (e.g., a touch sensor or a press button), comprising an input surface on the head-mounted device 116. The user input device 528 (e.g., a touch sensor or a press button) is used to receive input selections from a user for manipulating a GUI of the presented image.

[0109] Figure 5 Components of the head-mounted device 116 shown are located on one or more circuit boards (e.g., printed circuit boards (PCBs) or flexible PCBs) in the frame or temples. Alternatively or additionally, the depicted components may be located in the cheek pads, temple pieces, frame, hinges, or nose bridge of the head-mounted device 116. The left and right visible light imaging devices 506 may include digital imaging device elements, such as complementary metal-oxide-semiconductor (CMOS) image sensors, charge-coupled devices, camera lenses, or any other corresponding visible light or light-capturing elements that can be used to capture data, including images of scenes with unknown objects.

[0110] The head-mounted device 116 includes a memory 502 that stores instructions for performing a subset or all of the functions described herein. The memory 502 may also include a storage device.

[0111] like Figure 5 As shown, high-speed circuitry 526 includes a high-speed processor 530, memory 502, and high-speed wireless circuitry 532. In some examples, image display driver 520 is coupled to high-speed circuitry 526 and operated by high-speed processor 530 to drive the left and right image displays in image display 518. High-speed processor 530 can be any processor capable of managing high-speed communication and operation of any general-purpose computing system required by head-mounted device 116. High-speed processor 530 includes the processing resources required to manage high-speed data transmission to a wireless local area network (WLAN) over high-speed wireless connection 514 using high-speed wireless circuitry 532. In some examples, high-speed processor 530 executes the operating system of head-mounted device 116 (e.g., LINUX operating system) or other such operating system, and this operating system is stored in memory 502 for execution. Among other duties, high-speed processor 530, which executes the software architecture of head-mounted device 116, manages data transmission with high-speed wireless circuitry 532. In some examples, the high-speed wireless circuit 532 is configured to implement the Institute of Electrical and Electronics Engineers (IEEE) 802.11 communication standard, also referred to herein as WiFi. In some examples, the high-speed wireless circuit 532 can implement other high-speed communication standards.

[0112] The low-power wireless circuitry 524 and high-speed wireless circuitry 532 of the headset 116 may include a short-range transceiver (Bluetooth). TM The device 114 includes a wireless wide area network transceiver, a wireless local area network transceiver, or a wide area network transceiver (e.g., cellular or WiFi). The mobile device 114, including transceivers communicating via low-power wireless connection 512 and high-speed wireless connection 514, can be implemented using details of the architecture of the head-mounted device 116, and other elements of the network 516 can also be implemented in this way.

[0113] Memory 502 includes any storage device capable of storing various data and applications, including camera data generated by the left and right visible light cameras 506, the infrared camera 510, and the image processor 522, as well as images generated for display on an image display 518 via the image display driver 520. While memory 502 is shown as integrated with high-speed circuitry 526, in some examples, memory 502 may be a separate, independent component of the head-mounted device 116. In some such examples, electrical wiring may provide a connection from the image processor 522 or the low-power processor 536 to memory 502 via a chip including the high-speed processor 530. In some examples, the high-speed processor 530 may manage addressing of memory 502 such that the low-power processor 536 will activate the high-speed processor 530 whenever a read or write operation involving memory 502 is required.

[0114] like Figure 5 As shown, the low-power processor 536 or high-speed processor 530 of the head-mounted device 116 may be coupled to a camera device (visible light camera 506, infrared emitter 508 or infrared camera 510), an image display driver 520, a user input device 528 (e.g., a touch sensor or a press button), and a memory 502.

[0115] The head-mounted device 116 is connected to a host computer. For example, the head-mounted device 116 may pair with the mobile device 114 via a high-speed wireless connection 514 or connect to the server system 504 via a network 516. During pairing, the head-mounted device 116 may receive mirrored or other content from the mobile device 114 or the server system 504. The received content may be displayed on an image display 518 with two optical components. The server system 504 may be one or more computing devices as part of a service or network computing system, including, for example, a processor, memory, and a network communication interface for communicating with the mobile device 114 and the head-mounted device 116 via the network 516.

[0116] The mobile device 114 includes a processor and a network communication interface coupled to the processor. The network communication interface allows for communication over the network 516, the low-power wireless connection 512, or the high-speed wireless connection 514. The mobile device 114 can also store at least a portion of the instructions for generating binaural audio content in a memory of the mobile device 114 to implement the functionality described herein.

[0117] The output components of the head-mounted device 116 include visual components, such as displays (e.g., a liquid crystal display (LCD), a plasma display panel (PDP), a light-emitting diode (LED) display, a projector, or a waveguide). The image display of the optical assembly is driven by an image display driver 520. The output components of the head-mounted device 116 also include acoustic components (e.g., speakers), haptic components (e.g., a vibration motor), other signal generators, and the like. The input components (e.g., user input devices 528) of the head-mounted device 116, the mobile device 114, and the server system 504 can include alphanumeric input components (e.g., a keyboard, a touch screen configured to receive alphanumeric input, a photo-optical keyboard, or other alphanumeric input components), pointing components (e.g., a mouse, a touchpad, a trackball, a joystick, a motion sensor, or other pointing instrument), tactile input components (e.g., a physical button, a touch screen that provides location and force of touches or touch gestures, or other tactile input components), audio input components (e.g., a microphone), and the like.

[0118] The head-mounted device 116 can also include additional peripheral device elements. Such peripheral device elements can include biometric sensors, additional sensors, or display elements integrated with the head-mounted device 116. For example, the peripheral device elements can include any I / O components, including output components, motion components, positioning components, or any other such elements described herein.

[0119] For example, the biometric components include components for detecting expressions (e.g., hand expressions, facial expressions, vocal expressions, body gestures, or eye tracking), measuring biosignals (e.g., blood pressure, heart rate, body temperature, perspiration, or brain waves), identifying people (e.g., voice identification, retinal identification, facial identification, fingerprint identification, or electroencephalogram-based identification), and the like. The motion components include acceleration sensor components (e.g., an accelerometer), gravitation sensor components, rotation sensor components (e.g., a gyroscope), and the like. The positioning components include location sensor components (e.g., a GPS receiver component), Wi-Fi or Bluetooth® module components, or the like for generating location data. TMtransceiver, altitude sensor components (e.g., altimeter or barometer that detects air pressure from which altitude can be derived), orientation sensor components (e.g., magnetometer), etc. Such positioning system coordinates can also be received from the mobile device 114 via the low-power wireless circuit 534 or the high-speed wireless circuit 532 over the low-power wireless connection 512 and the high-speed wireless connection 514.

[0120] Reference is now made to Figure 6 . Figure 6 Aspects of a display card display mode are shown in accordance with some examples. A head 602 is shown wearing a head-mounted device 116 to view augmented content therein. The user can view content in a plurality of viewing directions 606. The viewing directions 606 can include a range of viewing directions 606 including, for example, a horizontal viewing direction 606, a vertically-upward viewing direction 606, a vertically-downward viewing direction 606, and a range of viewing directions 606 between these two extremes. Figure 6 An example viewing direction 606 is shown in

[0121] In some examples, the augmented content is displayed in an image display 518 of the head-mounted device 116. The augmented content is visible in a viewing pane 604 presented by the image display 518. The augmented content includes a virtual object 608. In the display card display mode shown, the viewing pane 604 is continuously aligned or oriented with a vertical axis 610 as shown. This mode can work well to present the augmented content in the image display 518, for example, when the user is viewing in a horizontal (or substantially horizontal) direction. Figure 8A An example view in this viewing direction 606 is provided in

[0122] However, this display card display mode can have the disadvantage that when the user (wearing the head-mounted device 116) looks upward and downward in such non-horizontal viewing directions 606 (e.g., 45 degrees from horizontal), the augmented content can tend to diminish, or even disappear in the extreme case when the user looks directly upward or directly downward (e.g., 90 degrees from horizontal upward or downward), when this display card display mode is continuously used to present the augmented content in the head-mounted device 116 in elevated or lowered viewing directions 606. Figure 8B and Figure 8C An example view in this regard is shown in

[0123] In Figure 8BIn the middle, for example, when the viewing direction 606 is at 45 degrees to the horizontal, the size of the viewing pane 604 is reduced. Here, as shown, at least the vertical dimension of the viewing pane 604 is reduced, i.e. in size. This reduction in size of the viewing pane 604 negatively impacts the content of the augmented content presented in the elevated or lowered viewing direction 606. For example, the virtual object 608 (the sun) presented in the viewing pane 604 can now only be seen as an elliptical or oblate object, which is non-realistic. In Figure 8C In the middle, for example, when the viewing direction 606 is at 45 degrees to the horizontal, the size of the viewing pane 604 is reduced. Here, as shown, at least the vertical dimension of the viewing pane 604 is reduced, i.e. in size. This reduction in size of the viewing pane 604 negatively impacts the content of the augmented content presented in the elevated or lowered viewing direction 606. For example, the virtual object 608 (the sun) presented in the viewing pane 604 can now only be seen as an elliptical or oblate object, which is non-realistic. In

[0124] Reference is now made to Figure 7 . Figure 7 Aspects of a head-locked display mode are shown in accordance with some examples. The head 602 is again shown as wearing the head-mounted device 116 to view augmented content therein. The user can view the content in a plurality of viewing directions 606. As in the previous, the viewing directions 606 can include a range of viewing directions 606 including, for example, a horizontal viewing direction 606, a vertically-up viewing direction 606, a vertically-down viewing direction 606 and a range of viewing directions 606 between these two extremes. Figure 7 An example viewing direction 606 is shown in the middle.

[0125] In some examples, the augmented content is displayed in the image display 518 of the head-mounted device 116. The augmented content is visible in a viewing pane 604 presented by the image display 518. The augmented content includes a virtual object 608. However, in the head-locked display mode shown, as shown, the viewing pane 604 is continuously circumferentially aligned around the user, or oriented with the circumferentially aligned axis 702. This head-locked display mode can work well to continuously orient the augmented content towards the user in any of the viewing directions 606. Figure 9 An example view in a given viewing direction 606 is provided in the middle. The virtual object 608 (in this case the sun emitting sunlight) can be seen as a circular object in the viewing pane 604 with full (or original) lateral and vertical dimensions maintained in any of the viewing directions 606. But when the display board display mode is continuously used in all viewing directions 606 to present the augmented content in the head-mounted device 116, the visual effect can be very unrealistic, unattractive, or in extreme cases even unpleasant for the user. In some cases, this effect is likened to wearing a television on one's head.

[0126] Reference is made to Figure 10Examples of the present disclosure seek to address the aforementioned disadvantage of persistently presenting augmented content using only one display mode, and to provide a hybrid display mode that includes two or more display modes within a range of viewing directions 606. In some examples, the hybrid display mode includes a mix or determined allocation of a card display mode and a headlock display mode based on a detected viewing direction 606 within the range.

[0127] In Figure 10 , the user's head 602 is again seen wearing the head-mounted device 116. The head-mounted device 116 includes an image display 518 in which augmented content including a virtual object 608 is displayed. The user's head 602 is laterally rotated and vertically rotated in the illustrated view to view the content over a range of viewing directions 606. As in the previous, the viewing directions 606 can include a range of viewing directions 606 including, for example, a horizontal viewing direction 606, a vertically upward viewing direction 606, a vertically downward viewing direction 606, and viewing directions 606 between these two extremes. In some examples, a detectable range of viewing directions 606 is determined. In some examples, a range 1006 of detectable vertical rotation of the user's head 602 is determined. The augmented content is visible in a viewing pane 604 presented by the image display 518. The augmented content includes a virtual object 608.

[0128] Referring to Figure 10 and Figure 11 , in operation 1102, the method 1100 of orienting content in an AR display determines an imaginary reference plane 1002 intersecting the user's head when the head-mounted device 116 is worn to view a virtual object 608 of augmented content visible in a viewing pane 604 having a vertical dimension and a lateral dimension when viewed in an image display 518 of the head-mounted device 116, where the imaginary reference plane 1002 coincides with a first viewing direction 1008 of the user's head 602. In operation 1104, the method 1100 of orienting content in an AR display determines or defines a range 1006 of detectable vertical rotation of the user's head 602.

[0129] In operation 1106, the method 1100 of orienting content in an AR display detects rotational movement of the user's head 602 in a vertical direction within the range 1006 of detectable vertical rotation when viewing the augmented content. In operation 1108, the method 1100 of orienting content in an AR display determines a second viewing direction 1010 of the user's head 602 when viewing the augmented content in the second viewing direction 1010 in response to the detected rotational movement.

[0130] In operation 1110, the method 1100 of orienting content in an AR display determines a reference angle 1004 between the hypothetical reference plane 1002 and the second viewing direction 1010, and assigns one or a combination of a billboard display mode and a headlock display mode to augmented content presented in the image display 518 of the head-mounted device 116 when viewed in the second viewing direction 1010 based on the reference angle 1004.

[0131] The method can further include wherein the hypothetical reference plane is horizontal and coincides with a horizontal dimension of the viewing pane. Other reference planes such as a vertical reference plane or a midrange reference plane are possible.

[0132] The method can further include wherein the detected rotational movement of the user’s head is detected using the head-mounted device.

[0133] The method can further include wherein assigning one or a combination of the billboard display mode and the headlock display mode to the augmented content is further based on a determined ratio of the billboard display mode to the headlock display mode within a range of detectable vertical rotation of the user’s head.

[0134] The method can further include wherein the determined ratio is directly proportional to the reference angle.

[0135] The method can further include wherein the determined ratio is algorithmically determined based on the reference angle.

[0136] The method can further include wherein algorithmically determining includes determining a first rotational elevation or declination of an axis normal to the viewing pane relative to the user’s head, and determining a second rotational elevation or declination of the axis normal to the viewing pane to make the viewing pane substantially orthogonal to the second viewing direction.

[0137] The method can further include wherein algorithmically determining further includes combining the first rotational angle and the second rotational angle into a virtual object orientation adjustment value, and applying the virtual object orientation adjustment value to the virtual object when displaying the augmented content in the second viewing direction.

[0138] The method can further include wherein applying the virtual object orientation adjustment value to the virtual object further includes maintaining a vertical dimension and a lateral dimension of the viewing pane when displaying the augmented content in the second viewing direction.

[0139] The method can further include continuously monitoring movement of the user’s head within the range of detectable vertical rotation, and dynamically recalculating the virtual object orientation adjustment value based on the continuously monitored movement of the user’s head.

[0140] The method can also include persistently maintaining the vertical dimension and the lateral dimension of the viewing pane while displaying the augmented content in other viewing directions. Other technical features can be readily apparent to one skilled in the art from the following figures, descriptions, and claims. The method can be embodied in computer-readable instructions for execution by one or more computer processors, such that the operations of the method can be performed, in part or in whole, by the content orientation system 234 executing on the head-mounted device 116, or the image processing system 202, or the image processor 522. Thus, the method 1100 of orienting content in an AR display is described herein by way of example with reference to an AR display. However, it should be appreciated that at least some of the operations of the method 1100 of orienting content in an AR display can be deployed on various other hardware configurations, and the method is not intended to be limited to the components or systems mentioned above. In embodiments, Figure 11 The operations described in the method 1100 correspond to the descriptions of at least the operations discussed above.

[0141] Examples

[0142] Some examples are provided in the present disclosure.

[0143] Example 1 includes a head-mounted device comprising: a processor; and a memory storing instructions that, when executed by the processor, configure the head-mounted device to: while the head-mounted device is worn to view a virtual object of augmented content that is viewable in a viewing pane having a vertical dimension and a lateral dimension when viewed in a display of the head-mounted device, determine a hypothetical reference plane that intersects a head of a user, the hypothetical reference plane coinciding with a first viewing direction of the head of the user; determine or define a range of detectable vertical rotation of the head of the user; within the range of detectable vertical rotation, detect a rotational movement of the head of the user in a vertical direction while viewing the augmented content; in response to the detected rotational movement, determine a second viewing direction of the head of the user while viewing the augmented content in the second viewing direction; and determine a reference angle between the hypothetical reference plane and the second viewing direction, and assign one or a combination of a display card display mode and a head-locked display mode to the augmented content that is presented in the display of the head-mounted device when viewed in the second viewing direction based on the reference angle.

[0144] Example 2 includes the elements of example 1, wherein the hypothetical reference plane is horizontal and coincides with a horizontal dimension of the viewing pane.

[0145] Example 3 includes the elements of example 2 or example 3, wherein the detected rotational movement of the head of the user is detected using the head-mounted device.

[0146] Example 4 includes elements of any of examples 1-3, wherein assigning one or a combination of a billboard display mode and a headlock display mode to the augmented content is further based on a determined ratio of the billboard display mode to the headlock display mode within a range of detectable vertical rotation of the user's head.

[0147] Example 5 includes elements of any of examples 1-4, wherein the determined ratio is directly proportional to a reference angle.

[0148] Example 6 includes elements of any of examples 1-5, wherein the determined ratio is algorithmically determined based on a reference angle.

[0149] Example 7 includes elements of any of examples 1-6, wherein algorithmically determining comprises: determining a first rotational elevation or declination of an axis normal to the viewing pane relative to the user's head; and determining a second rotational elevation or declination of the axis normal to the viewing pane to make the viewing pane substantially orthogonal to the second viewing direction.

[0150] Example 8 includes elements of any of examples 1-7, wherein algorithmically determining further comprises: combining the first and second rotational angles into a virtual object orientation adjustment value; and applying the virtual object orientation adjustment value to the virtual object when displaying the augmented content in the second viewing direction.

[0151] Example 9 includes elements of any of examples 1-8, wherein applying the virtual object orientation adjustment value to the virtual object further comprises: maintaining a vertical dimension and a horizontal dimension of the viewing pane when displaying the augmented content in the second viewing direction.

[0152] Example 10 includes elements of any of examples 1-9, wherein the instructions further configure the device to: continuously monitor movement of the user's head within the range of detectable vertical rotation; and dynamically recalculate the virtual object orientation adjustment value based on the continuously monitored movement of the user's head.

[0153] Example 11 includes elements of any of examples 1-10, wherein the instructions further configure the device to: continuously maintain a vertical dimension and a horizontal dimension of the viewing pane when displaying the augmented content in other viewing directions.

[0154] Machine architecture

[0155] Figure 12is a diagrammatic representation of the machine 1200 in which instructions 1202 (e.g., software, a program, an application, an applet, an app, or other executable code) for causing the machine 1200 to perform any one or more of the methodologies discussed herein can be executed. For example, the instructions 1202 can cause the machine 1200 to execute any one or more of the methods described herein. The instructions 1202 transform the general, non-programmed machine 1200 into a particular machine 1200 programmed to carry out the described and illustrated functions in the manner described. The machine 1200 can operate as a standalone device or can be coupled (e.g., networked) to other machines. In a networked deployment, the machine 1200 can operate in the capacity of a server machine or a client machine in server-client network environments, or as a peer machine in peer-to-peer (or distributed) network environments. The machine 1200 can comprise, but not be limited to, a server computer, a client computer, a personal computer (PC), a tablet computer, a laptop computer, a netbook, a set-top box (STB), a personal digital assistant (PDA), an entertainment media system, a cellular telephone, a smart phone, a mobile device, a wearable device (e.g., a smart watch), a smart home device (e.g., a smart appliance), other smart devices, a web appliance, a network router, a network switch, a network bridge, or any machine capable of executing the instructions 1202, sequentially or otherwise, that specify actions to be taken by machine 1200. Further, while only a single machine 1200 is illustrated, the term “machine” shall also be taken to include a collection of machines that individually or jointly execute the instructions 1202 to perform any one or more of the methodologies discussed herein. For example, the machine 1200 can include the user system 102 or any of the multiple server devices that form part of the interaction server system 110. In some examples, the machine 1200 can further include both a client system and a server system, wherein certain operations of a particular method or algorithm are performed in the server side and certain operations of the particular method or algorithm are performed in the client side.

[0156] The machine 1200 can include processors 1204, memory 1206, and input / output (I / O) components 1208, which can be configured to communicate with each other via a bus 1210. In an example, the processors 1204 (e.g., a central processing unit (CPU), a reduced instruction set computing (RISC) processor, a complex instruction set computing (CISC) processor, a graphics processing unit (GPU), a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a radio-frequency integrated circuit (RFIC), another processor, or any suitable combination thereof) can include, for example, a processor 1212 and a processor 1214 that execute instructions 1202. The term “processor” is intended to include multiple cores of a single processor, multiple processors, or any combination thereof. For example, the processors 1204 can include a single core processor, dual core processor, quad core processor, or any suitable combination of processors. By way of another example, the processors 1204 can include a single thread processor, a multi-threaded processor, or any suitable combination thereof. The term “memory” is intended to include single or multiple storage units, which can be of any suitable type including, for example, volatile memory (e.g., random access memory (RAM)), non-volatile memory (e.g., read-only memory (ROM), magnetic storage, optical storage, or any suitable combination thereof). The memory 1206 can include a main memory 1216, a static memory 1218, and a storage unit 1220, each of which can be accessed via the bus 1210 by the processors 1204. The main memory 1206, the static memory 1218, and the storage unit 1220 store the instructions 1202 that implement any one or more of the methods or functions described herein. The instructions 1202 can further reside completely, or partially, within the main memory 1216, within the static memory 1218, within the storage unit 1220, within at least one of the processors 1204 (e.g., within the cache memory of the processors), or any suitable combination thereof, during their execution by the machine 1200. Figure 12 Although illustrated with multiple processors 1204, the machine 1200 can include a single processor with a single core, a single processor with multiple cores (e.g., a multi-core processor), multiple processors with a single core, multiple processors with multiple cores, or any combination thereof.

[0157] The memory 1206 includes a main memory 1216, a static memory 1218, and a storage unit 1220, each of which can be accessed via the bus 1210 by the processors 1204. The main memory 1206, the static memory 1218, and the storage unit 1220 store the instructions 1202 that implement any one or more of the methods or functions described herein. The instructions 1202 can further reside completely, or partially, within the main memory 1216, within the static memory 1218, within the storage unit 1220, within at least one of the processors 1204 (e.g., within the cache memory of the processors), or any suitable combination thereof, during their execution by the machine 1200.

[0158] The I / O components 1208 can include various components for receiving input, providing output, producing output, transmitting information, exchanging information, capturing measurements, and so on. The specific I / O components 1208 included in the machine 1200 will depend on the type of machine. For example, portable machines such as mobile phones will likely include a touch input device or other such input mechanisms, while a headless server machine will not include such a touch input device. It will be appreciated that the I / O components 1208 can include Figure 12Many other components not shown in FIG. 12 can also be included. In various examples, the I / O components 1208 can include user output components 1224 and user input components 1226. The user output components 1224 can include visual components (e.g., a display such as a PDP, an LED display, an LCD, a projector, or a cathode ray tube (CRT)), acoustic components (e.g., speakers), haptic components (e.g., a vibratory motor, resistance mechanisms), other signal generators, and so forth. The user input components 1226 can include alphanumeric input components (e.g., a keyboard, a touch screen configured to receive alphanumeric input, a photocopier, or other alphanumeric input components), pointing components (e.g., a mouse, a touchpad, a trackball, a joystick, a motion sensor, or other pointing instruments), tactile input components (e.g., a physical button, a touch screen that provides location and force of touches or touch gestures, or other tactile input components), audio input components (e.g., a microphone), and the like.

[0159] In further examples, the I / O components 1208 can include biometric components 1228, motion components 1230, environmental components 1232, or positioning components 1234, among a myriad of other components. For example, the biometric components 1228 include components to detect expressions (e.g., hand expressions, facial expressions, vocal expressions, body gestures, or eye tracking), measure biosignals (e.g., blood pressure, heart rate, body temperature, perspiration, or brain waves), identify a person (e.g., voice identification, retinal identification, facial identification, fingerprint identification, or electroencephalogram-based identification), and the like.

[0160] The environmental components 1232 include, for example, one or more cameras (with still and / or video capabilities), illumination sensors (e.g., photometers), temperature sensors (e.g., one or more thermometers that detect ambient temperature), humidity sensors, pressure sensors (e.g., barometers), acoustic sensors (e.g., one or more microphones that detect background noise), proximity sensors (e.g., infrared sensors that detect nearby objects), gas sensors (e.g., gas detection sensors to detect concentrations of hazardous gases for safety or to measure pollutants in ambient air), or other components that can provide indications, measurements, or signals corresponding to a physical environment.

[0161] With respect to cameras, the user system 102 can have a camera system that includes a front-facing camera, e.g., on a front surface of the user system 102, and a rear-facing camera on a rear surface of the user system 102. The front-facing camera can be used, for example, to capture still images and videos of a user of the user system 102 (e.g., “selfies”), which can then be augmented with the above-described augmentation data (e.g., filters). The rear-facing camera can be used, for example, to capture still images and videos in a more conventional camera mode, which are similarly augmented with augmentation data. In addition to front- and rear-facing cameras, the user system 102 can include a 360° camera for capturing 360° photos and videos.

[0162] In addition, the camera system of the user system 102 can include dual rear-facing cameras (e.g., a primary camera and a depth-sensing camera), or even triple, quad, or penta- rear-facing camera configurations on the front and rear sides of the user system 102. These multi-camera systems can include, for example, a wide-angle camera, an ultra-wide-angle camera, a telephoto camera, a macro camera, and a depth sensor.

[0163] The positioning component 1234 includes location sensor components (e.g., GPS receiver components), altitude sensor components (e.g., altimeters or barometers that detect air pressure from which altitude can be derived), orientation sensor components (e.g., magnetometers), and so on.

[0164] Communication can be implemented using a wide variety of technologies. The I / O component 1208 also includes a communication component 1236, which can be operable to couple (via a Components (e.g., Low power consumption), Components and other communication components for providing communication via other modalities. The device 1240 can be another machine or any of a wide variety of peripheral devices (e.g., a peripheral device coupled via USB).

[0165] Furthermore, communication component 1236 can detect identifiers, or includes components operable to detect identifiers. For example, communication component 1236 may include a radio frequency identification (RFID) tag reader component, an NFC smart tag detection component, an optical reader component (e.g., for detecting one-dimensional barcodes such as Universal Product Code (UPC) barcodes, QR codes such as Quick Response (QR) codes, Aztec codes, data matrices, and data symbols). TM This includes optical sensors for multidimensional barcodes and other optical codes such as MaxiCode, PDF417, UltraCode, and UCC RSS-2D barcodes, or acoustic detection components (e.g., microphones for identifying audio signals of the tags). Additionally, various information can be obtained via communication component 1236, such as location via Internet Protocol (IP) geolocation, etc. Location can be determined by signal triangulation or by detecting NFC beacon signals that indicate a specific location.

[0166] Various memories (e.g., main memory 1216, static memory 1218, and the memory of processor 1204) and storage units 1220 may store one or more sets of instructions and data structures (e.g., software) implemented or used by any one or more of the methods or functions described herein. These instructions (e.g., instruction 1202) cause various operations to implement the disclosed examples when executed by processor 1204.

[0167] Instruction 1202 can be sent or received over network 1238 via a transmission medium using a network interface device (e.g., a network interface component included in communication component 1236) and using any of several known transmission protocols (e.g., HTTP). Similarly, instruction 1202 can be sent or received via a transmission medium coupled to device 1240 (e.g., peer-to-peer coupling).

[0168] Software Architecture

[0169] Figure 13is a block diagram 1300 illustrating software architecture 1302, which can be installed on any one or more of the devices described herein. The software architecture 1302 is supported by hardware of a machine, such as machine 1304 that includes processors 1306, memory 1308, and I / O components 1310. In this example, the software architecture 1302 can be conceptualized as a stack of layers, where each layer provides particular functionality. The software architecture 1302 includes layers such as an operating system 1312, libraries 1314, frameworks 1316, and applications 1318. Operationally, the applications 1318 make API calls 1320 through the software stack and receive messages 1322 in response to the API calls 1320.

[0170] The operating system 1312 manages hardware resources and provides common services. The operating system 1312 includes, for example, a kernel 1324, services 1326, and drivers 1328. The kernel 1324 acts as an abstraction layer between the hardware and the other software layers. For example, the kernel 1324 provides memory management, processor management (e.g., scheduling), component management, networking, and security settings, among other functionality. The services 1326 can provide other common services that the applications 1318 and other software layers use. The drivers 1328 are responsible for controlling or interfacing with the underlying hardware, according to the implementations. For instance, the drivers 1328 can include display drivers, camera drivers, Bluetooth® drivers, flash or low energy drivers, flash drivers, serial communication drivers (e.g., USB drivers), drivers, audio drivers, power management drivers, and the like.

[0171] The libraries 1314 provide a higher-level common infrastructure that can be used by the applications 1318 and others. The libraries 1314 can include system libraries 1330 (e.g., C standard library) that provide functions such as memory allocation functions, string manipulation functions, mathematical functions, and the like. In addition, the libraries 1314 can include API libraries 1332 such as media libraries (e.g., libraries to support presentation and manipulation of various media formats such as Moving Picture Experts Group-4 (MPEG4), Advanced Video Coding (H.264 or AVC), Moving Picture Experts Group Layer-3 (MP3), Advanced Audio Coding (AAC), Adaptive Multi-Rate (AMR) audio codec, Joint Photographic Experts Group (JPEG or JPG), or Portable Network Graphics (PNG)), graphics libraries (e.g., an OpenGL framework used to render 2D and 3D in graphic content on a display), database libraries (e.g., SQLite to provide various relational database functions), web libraries (e.g., WebKit to provide web browsing functionality), and the like. The libraries 1314 also include a wide variety of other libraries 1334 to provide many other APIs to the applications 1318.

[0172] Framework 1316 provides common high-level infrastructure for use by application 1318. For example, framework 1316 provides various GUI functions, high-level resource management, and high-level location services. Framework 1316 can provide a wide range of other APIs that can be used by application 1318, some of which may be specific to a particular operating system or platform.

[0173] In the example, application 1318 may include a home application 1336, a contacts application 1338, a browser application 1340, a book reader application 1342, a location application 1344, a media application 1346, a messaging application 1348, a game application 1350, and a wide variety of other applications such as third-party application 1352. Application 1318 is a program that performs the functions defined in the program. One or more applications of application 1318 can be created using various programming languages, such as object-oriented programming languages ​​(e.g., Objective-C, Java, or C++) or procedural programming languages ​​(e.g., C or assembly language). In a particular example, third-party application 1352 (e.g., an entity other than the vendor of a particular platform using Android) TM or iOS TM Applications developed using the SDK can be used in systems such as iOS. TM ANDROID TM , Mobile software running on the phone's mobile operating system or other mobile operating systems. In this example, a third-party application 1352 can activate API call 1320 provided by the operating system 1312 to facilitate the functionality described herein.

[0174] Glossary

[0175] "Carrier signal" refers to any intangible medium, such as a medium capable of storing, encoding, or carrying instructions to be executed by a machine and including digital or analog communication signals, or other intangible medium that facilitates the communication of such instructions. Instructions can be sent or received over a network using a transmission medium via a network interface device.

[0176] "Client device" means any machine that interfaces with a communication network to obtain resources from one or more server systems or other client devices. Client devices can be, but are not limited to, mobile phones, desktop computers, laptop computers, PDAs, smartphones, tablet computers, ultrabooks, netbooks, laptop computers, multiprocessor systems, microprocessor-based or programmable consumer electronics, game consoles, STBs, or any other communication device that a user can use to access the network.

[0177] "Communication network" refers to one or more parts of a network, such as an ad hoc network, intranet, extranet, virtual private network (VPN), local area network (LAN), WLAN, wide area network (WAN), wireless WAN (WWAN), metropolitan area network (MAN), the Internet, a part of the Internet, a part of the Public Switched Telephone Network (PSTN), a POTS (Plain Old-Style Telephone Service) network, a cellular telephone network, a wireless network, etc. A network, other types of networks, or a combination of two or more such networks. For example, a network or part of a network may include a wireless network or a cellular network, and the coupling may be a Code Division Multiple Access (CDMA) connection, a Global System for Mobile Communications (GSM) connection, or other types of cellular or wireless coupling. In this example, the coupling can implement any data transmission technology of various types, such as Single Carrier Radio Transmission (1xRTT), Evolved Data Optimization (EVDO), General Packet Radio Service (GPRS), Enhanced Data Rate Evolution of GSM (EDGE), the 3rd Generation Partnership Project (3GPP) including 3G, fourth-generation wireless (4G) networks, Universal Mobile Telecommunications System (UMTS), High-Speed ​​Packet Access (HSPA), Global Microwave Access Interoperability (WiMAX), Long Term Evolution (LTE) standards, other data transmission technologies defined by various standards setting organizations, other long-distance protocols, or other data transmission technologies.

[0178] A "component" refers to a logical, physical, or device that has boundaries that are defined by function or subroutine calls, branches points, APIs, or other techniques that provide a partitioning or modularization of a particular processing or control function. Components can be combined via their interfaces to create a machine process. A component can be a packaged functional hardware unit designed for use with other components and typically provides a particular function or set of functions. Components can be software programs that are executed on a machine or hardware components that implement a specific function. A "hardware component" is a tangible unit that can be employed in a system, and can be a specialized machine or machine part, and / or can be a specialized portion of a machine. In various examples, a hardware component can include a specialized processor, or a portion thereof, that has been programmed to implement a specific function or a group of functions. A hardware component can be a general-purpose processor, such as a central processing unit (CPU) or a general-purpose graphics processing unit (GPGPU), that has been programmed with specific component firmware to implement a specific function or a group of functions. A hardware component can be caused to be formed from configuration of general- purpose hardware (e.g., by configuration of general purpose processing hardware with specific component firmware). In some examples, a hardware component can be located on a single chassis, while in other examples a hardware component can be distributed across multiple, non- contiguous parts or components. A hardware component can be a specialized component, such as an application-specific integrated circuit (ASIC) or a field-programmable gate array (FPGA). In some examples, a hardware component can include processor firmware that has been programmed to carry out a predetermined logic function or a portion of a predetermined logic function. The software can include, but is not limited to, a real-time operating system, APIs, drivers, kernel, processes, libraries, modules, firmware, and / or any other software that can be executed on a machine.Where multiple hardware components are present, communication can be achieved through signal transmission among or between two or more of the hardware components (e.g., through appropriate circuits and buses). In examples where multiple hardware components are configured or instantiated at different times, communication between such hardware components can for example be achieved by storing information in memory structures to which the multiple hardware components have access, and by retrieving information from the memory structures. For example, one hardware component can perform an operation and store output of that operation in a memory device to which it is communicatively coupled. A further hardware component can then, at a later time, access the memory device to retrieve and process the stored output. The hardware components can also initiate communications with input or output devices, and can operate on a resource (e.g., a collection of information). The various operations of example methods described herein can be performed, at least partially, by one or more processors that are temporarily configured (e.g., by software) or permanently configured to perform the relevant operations. Whether temporarily or permanently configured, such processors can constitute processor-implemented components that operate to perform one or more operations or functions described herein. As used herein, “processor-implemented component” refers to a hardware component implemented using one or more processors. Similarly, the methods described herein can be at least partially processor-implemented, with a particular processor or processors being an example of hardware. For example, at least some of the operations of a method can be performed by one or more processors or processor-implemented components. Moreover, a processor or processors can also operate to support performance of the relevant operations in a “cloud computing” environment or as a “software as a service” (SaaS). For example, at least some of the operations can be performed by a group of computers (as examples of machines including processors), with these operations being accessible via a network (e.g., the Internet) and via one or more appropriate interfaces (e.g., an API). The performance of certain of the operations can be distributed among the processors, not only residing within a single machine, but deployed across a number of machines. In some examples, the processors or processor-implemented components can be located in a single geographic location (e.g., within a home environment, an office environment, or a server farm). In other examples, the processors or processor-implemented components can be distributed across a number of geographic locations.

[0179] Some examples described herein include reference to terms such as “horizontal” and “vertical.” In some examples, these terms include “substantially” horizontal and / or vertical, as appropriate, and are not, for example, strictly “absolutely vertical” or “absolutely horizontal.” Some examples include, within their scope, horizontal or vertical that is plus or minus 5 degrees, plus or minus 10 degrees, or plus or minus 15 degrees from horizontal or vertical, respectively. Some examples include, within their scope, horizontal or vertical that is in a range of 5 to 45 degrees from horizontal or vertical, respectively.

[0180] “Computer-readable storage medium” refers to both machine-storage media and transmission media, e.g., both storage devices / media and carrier waves / modulated data signals. Accordingly, the terms “machine-readable medium,” “computer-readable medium,” and “device-readable medium” mean the same thing and can be used interchangeably in this disclosure.

[0181] “Ephemeral message” refers to a message that is accessible, e.g., for a time-limited duration. An ephemeral message can be text, image, video, etc. The access time for an ephemeral message can be set by the message sender. Alternatively, the access time can be a default setting or a setting specified by the recipient. Regardless of the setting technique, the message is temporary.

[0182] “Machine-storage medium” refers to a single or multiple storage devices and media (e.g., a centralized or distributed database, and associated caches and servers) that store executable instructions, routines, and data. Thus, the term should be taken to include, but not be limited to, solid state memory, as well as optical and magnetic media, including memory internal or external to a processor. Specific examples of machine-storage media, computer-storage media, and device-storage media include non-volatile memory, including, e.g., semiconductor memory devices, e.g., Erasable Programmable Read-Only Memories (EPROM), Electrically Erasable Programmable Read-Only Memories (EEPROM), FPGAs, and flash memory devices; magnetic disks such as internal hard disks and removable disks; magneto-optical disks; and CD-ROM and DVD-ROM disks. The terms “machine-storage medium,” “device-storage medium,” and “computer-storage medium” mean the same thing and can be used interchangeably in this disclosure. The terms “machine-storage medium,” “computer-storage medium,” and “device-storage medium” explicitly exclude carrier waves, modulated data signals, and other such media, at least some of which are, instead, included in the term “signal medium.”

[0183] “Non-transitory computer-readable storage medium” refers to, e.g., a tangible medium that is capable of storing, encoding, or carrying the instructions for execution by a machine.

[0184] “Signal medium” refers to, e.g., any intangible medium that is capable of storing, encoding, or carrying the instructions for execution by a machine and includes digital or analog communications signals or other intangible media to facilitate communication of software or data. The term “signal medium” shall be taken to include any form of a modulated data signal, carrier wave, and so on. The term “modulated data signal” means a signal that has one or more of its characteristics set or changed in such a manner as to encode information in the signal. The terms “transmission medium” and “signal medium” mean the same thing and can be used interchangeably in this disclosure.

[0185] “User device” refers to a device that is, for example, accessed, controlled, or owned by a user, and with which the user interacts to perform actions or interactions with other users or computer systems.

Claims

1. A method for orienting content in an augmented reality display in a head-mounted device, the method comprising: When wearing the head-mounted device to view virtual objects of enhanced content that are visible in a viewing pane having vertical and horizontal dimensions when viewed on the display of the head-mounted device, an imaginary reference plane intersecting with the user's head is determined, the imaginary reference plane being aligned with a first viewing direction of the user's head; Determine or limit the detectable range of vertical rotation of the user's head; Within the detectable vertical rotation range, the vertical rotational movement of the user's head is detected while viewing the enhanced content; In response to detected rotational movement, when viewing the enhanced content in a second viewing direction, the second viewing direction of the user's head is determined; as well as A reference angle is determined between the hypothetical reference plane and the second viewing direction, and based on the reference angle, one or a combination of a display mode and a headlock display mode is assigned to the enhanced content presented on the display of the head-mounted device when viewed in the second viewing direction.

2. The method according to claim 1, wherein, The imaginary reference plane is horizontal and has the same horizontal dimension as the viewing pane.

3. The method according to claim 1, wherein, The detected rotational movement of the user's head is detected using the head-mounted device.

4. The method according to claim 1, wherein, Assigning one or a combination of a display mode and a headlock display mode to the enhanced content is also based on a determined ratio of the display mode to the headlock display mode within the range of detectable vertical rotation of the user's head.

5. The method according to claim 4, wherein, The determined ratio is proportional to the reference angle.

6. The method according to claim 4, wherein, The determined ratio is algorithmically determined based on the reference angle.

7. The method according to claim 6, wherein, The algorithmic determination includes: Determine a first rotational elevation or tilt angle relative to the user's head, perpendicular to the axis of the viewing window; and Determine a second rotation angle or tilt angle perpendicular to the axis of the viewing window so that the viewing window is substantially orthogonal to the second viewing direction.

8. The method according to claim 7, wherein, The algorithmic determination also includes: Combine the first rotation angle and the second rotation angle to form a virtual object orientation adjustment value; and When the enhanced content is displayed in the second viewing direction, the virtual object orientation adjustment value is applied to the virtual object.

9. The method according to claim 8, wherein, Applying the virtual object orientation adjustment value to the virtual object further includes maintaining the vertical and horizontal dimensions of the viewing pane when the enhanced content is displayed in the second viewing direction.

10. The method of claim 9, further comprising: The movement of the user's head is continuously monitored within the detectable range of vertical rotation; as well as The virtual object orientation adjustment value is dynamically recalculated based on the continuously monitored head movements of the user.

11. The method of claim 9, further comprising: When the enhanced content is displayed in other viewing directions, the vertical and horizontal dimensions of the viewing pane are maintained.

12. A head-mounted device, comprising: processor; as well as A memory that stores instructions, when executed by the processor, to configure the head-mounted device to: When wearing the head-mounted device to view virtual objects of enhanced content that are visible in a viewing pane having vertical and horizontal dimensions when viewed on the display of the head-mounted device, an imaginary reference plane intersecting with the user's head is determined, the imaginary reference plane being aligned with a first viewing direction of the user's head; Determine or limit the detectable range of vertical rotation of the user's head; Within the detectable vertical rotation range, the vertical rotational movement of the user's head is detected while viewing the enhanced content; In response to detected rotational movement, when viewing the enhanced content in a second viewing direction, the second viewing direction of the user's head is determined; as well as A reference angle is determined between the hypothetical reference plane and the second viewing direction, and based on the reference angle, one or a combination of a display mode and a headlock display mode is assigned to the enhanced content presented on the display of the head-mounted device when viewed in the second viewing direction.

13. The head-mounted device according to claim 12, wherein, The imaginary reference plane is horizontal and has the same horizontal dimension as the viewing pane.

14. The head-mounted device according to claim 12, wherein, The detected rotational movement of the user's head is detected using the head-mounted device.

15. The head-mounted device according to claim 12, wherein, Assigning one or a combination of a display mode and a headlock display mode to the enhanced content is also based on a determined ratio of the display mode to the headlock display mode within the range of detectable vertical rotation of the user's head.

16. The head-mounted device according to claim 15, wherein, The determined ratio is proportional to the reference angle.

17. The head-mounted device according to claim 15, wherein, The determined ratio is algorithmically determined based on the reference angle.

18. The head-mounted device according to claim 17, wherein, The algorithmic determination includes: Determine a first rotational elevation or tilt angle relative to the user's head, perpendicular to the axis of the viewing window; and Determine a second rotation angle or tilt angle perpendicular to the axis of the viewing window so that the viewing window is substantially orthogonal to the second viewing direction.

19. The head-mounted device according to claim 18, wherein, The algorithmic determination also includes: Combine the first rotation angle and the second rotation angle to form a virtual object orientation adjustment value; and When the enhanced content is displayed in the second viewing direction, the virtual object orientation adjustment value is applied to the virtual object.

20. The head-mounted device according to claim 19, wherein, Applying the virtual object orientation adjustment value to the virtual object further includes maintaining the vertical and horizontal dimensions of the viewing pane when the enhanced content is displayed in the second viewing direction.

21. The head-mounted device according to claim 20, wherein, The instructions also configure the device as follows: Continuously monitor the user's head movement within the detectable vertical rotation range; and The virtual object orientation adjustment value is dynamically recalculated based on the continuously monitored head movements of the user.

22. The head-mounted computing device according to claim 20, wherein, The instructions also configure the device to: When the enhanced content is displayed in other viewing directions, the vertical and horizontal dimensions of the viewing pane are maintained.

23. A non-transitory computer-readable storage medium, the computer-readable storage medium comprising instructions that, when executed by a computer, cause the computer to: When wearing the head-mounted device to view virtual objects of enhanced content that are visible in a viewing pane having vertical and horizontal dimensions when viewed on the display of the head-mounted device, an imaginary reference plane intersecting with the user's head is determined, the imaginary reference plane being aligned with a first viewing direction of the user's head; Determine or limit the detectable range of vertical rotation of the user's head; Within the detectable vertical rotation range, the vertical rotational movement of the user's head is detected while viewing the enhanced content; In response to detected rotational movement, when viewing the enhanced content in a second viewing direction, the second viewing direction of the user's head is determined; as well as A reference angle is determined between the hypothetical reference plane and the second viewing direction, and based on the reference angle, one or a combination of a display mode and a headlock display mode is assigned to the enhanced content presented on the display of the head-mounted device when viewed in the second viewing direction.

24. The computer-readable storage medium according to claim 23, wherein, The imaginary reference plane is horizontal and has the same horizontal dimension as the viewing pane.

25. The computer-readable storage medium according to claim 23, wherein, The detected rotational movement of the user's head is detected using the head-mounted device.

26. The computer-readable storage medium according to claim 23, wherein, Assigning one or a combination of a display mode and a headlock display mode to the enhanced content is also based on a determined ratio of the display mode to the headlock display mode within the range of detectable vertical rotation of the user's head.

27. The computer-readable storage medium according to claim 26, wherein, The determined ratio is proportional to the reference angle.

28. The computer-readable storage medium according to claim 26, wherein, The determined ratio is algorithmically determined based on the reference angle.

29. The computer-readable storage medium according to claim 28, wherein, The algorithmic determination includes: Determine a first rotational elevation or tilt angle relative to the user's head, perpendicular to the axis of the viewing window; and Determine a second rotation angle or tilt angle perpendicular to the axis of the viewing window so that the viewing window is substantially orthogonal to the second viewing direction.

30. The computer-readable storage medium according to claim 29, wherein, The algorithmic determination also includes: Combine the first rotation angle and the second rotation angle to form a virtual object orientation adjustment value; and When the enhanced content is displayed in the second viewing direction, the virtual object orientation adjustment value is applied to the virtual object.

31. The computer-readable storage medium according to claim 30, wherein, Applying the virtual object orientation adjustment value to the virtual object further includes maintaining the vertical and horizontal dimensions of the viewing pane when the enhanced content is displayed in the second viewing direction.

32. The computer-readable storage medium according to claim 31, wherein, The instructions also configure the computer to: Continuously monitor the user's head movement within the detectable vertical rotation range; and The virtual object orientation adjustment value is dynamically recalculated based on the continuously monitored head movements of the user.

33. The computer-readable storage medium according to claim 31, wherein, The instructions also configure the computer to: When the enhanced content is displayed in other viewing directions, the vertical and horizontal dimensions of the viewing pane are maintained.

Images