Preventing UI occlusion in VR environments

Abstract

The present disclosure relates to preventing user interface occlusion in a virtual reality environment. In one embodiment, a client system receives a request for an application to access a user interface element in a virtual reality environment, and generates the user interface element in the virtual reality environment. The user interface element at the generated location occludes a portion of the application. The client system detects an input to move the user interface element from one location in the virtual reality environment to another location in the virtual reality environment relative to a display area of ​​the application, and stores location data associated with the other location in the virtual reality environment in a location database. The location data is associated with the application. The client system receives another request for the application to access the user interface element in the virtual reality environment, and generates the user interface element in the virtual reality environment based on the location data associated with the application.

Description

Technical Field

[0001] The present disclosure generally relates to controls and interfaces for user interaction and experience in virtual reality environments. Background Art

[0002] Virtual reality is a computer-generated simulation of an environment (e.g., a 3D environment) with which a user can interact in a seemingly real or physical manner. A virtual reality system, which may be a single device or a group of devices, can generate the simulation to be displayed to a user, for example, on a virtual reality headset or some other display device. The simulation may include images, sounds, tactile feedback, and / or other sensations that simulate a real or imagined environment. As virtual reality becomes more and more prominent, its useful range of applications is rapidly expanding. The most common applications of virtual reality include games or other interactive content, but other applications are not far behind, such as viewing visual media items (e.g., photos, videos) for entertainment or training purposes. The feasibility of using virtual reality to simulate real-life conversations and other user interactions is also being explored. Summary of the invention

[0003] Disclosed herein are a number of different ways to render and interact with a virtual (or augmented) reality environment. A virtual reality system may render a virtual environment that may include a virtual space rendered for display to one or more users. Users may view and interact within this virtual space and the broader virtual environment in any suitable manner. One objective of the disclosed method is to improve the safety of the virtual environment. In a particular embodiment, a virtual reality system may provide a method for distinguishing operating system (OS) content generated within a virtual reality environment from third-party content generated and selectively displaying the content in different display planes accordingly. As an example and not by way of limitation, operating system content may include various processes, such as system updates or other processes run by the OS. As another example and not by way of limitation, third-party content may involve content generated by applications running on the virtual reality system, such as games and interactive content. As another example and not by way of limitation, an OS user interface (e.g., a keyboard, menu, pop-up window, or any other user interface generated by the operating system) may be distinguished from a third-party user interface (e.g., a keyboard, menu, pop-up window, or any other user interface generated by a third party). In a virtual reality environment, a user should be able to trust elements generated in the virtual reality environment, such as a third-party application that the user is currently connected to. However, there may be malicious third-party entities that attempt to compromise the user's security and obtain the user's confidential information without authorization. As an example and not by way of limitation, a malicious third-party entity may generate a user interface element, such as a keyboard, that is disguised as part of the system software, hoping that the user will use it to enter confidential information (e.g., passwords, social security numbers, etc.). Despite the presence of malicious third-party entities, user 101 may rely on and maintain use of system-generated content, such as user interface elements (e.g., keyboards).

[0004] To combat third parties with malicious intent, in certain embodiments, the virtual reality system may accept any request to access a user interface element (e.g., a keyboard) and determine whether the requested user interface element is a system user interface element generated by the OS or a third-party user interface element generated by a third-party application or based on third-party content. As an example and not by way of limitation, a request may be received from a user interacting with a third-party application to generate a user interface element (e.g., a keyboard) to enter his or her credentials into an input field. In certain embodiments, the virtual reality system may accept the request and determine whether the user interface element to be called is a user interface of the system. In certain embodiments, in order to distinguish between user interface elements generated by a third-party application and user interface elements generated by the OS, the virtual reality system may generate these two different user interface elements in two separate planes. As an example and not by way of limitation, the virtual reality system may determine that the user requests to enter his or her credentials to approve a software update for the OS, and display the system keyboard on a plane at a distance from the user in the virtual reality environment. As another example and not by way of limitation, the virtual reality system may receive a request to display a third-party keyboard for entering user credentials for a service, and display the keyboard in a plane at a distance from the user in the virtual reality environment, wherein the second plane generated for the third-party keyboard may be farther from the user than the first plane generated for the OS content. Other differences between OS-generated user interface elements and third-party applications can include the orientation of the user interface element (e.g., which direction the keyboard is facing). In certain embodiments, the first plane can be dedicated to user interface elements generated for the OS, and the second plane can be dedicated to user interface elements generated by third-party applications to further distinguish between user interface elements generated by the OS or third-party applications.

[0005] In certain embodiments, the virtual reality system can further improve system security by limiting data access to applications that have been specifically authorized. Many users can perform a certain degree of multitasking and open and run several applications at any given time. As an example and not by way of limitation, a user can check and respond to urgent emails while watching a movie or playing a game. In certain embodiments, these applications can utilize sensor data (e.g., accelerometer data, gyroscope data, magnetometer data, eye tracking data, etc.) received from the VR headset to perform application-related functions (e.g., changing the field of view of the user moving his or her head). Therefore, if the user switches between applications, the previous application may still be receiving sensor data. In the case where the user enters confidential information in the current application, this may compromise the user's security because an entity can obtain sensor data (e.g., where the user is looking) to determine which characters are entered into the keyboard in the VR environment with some degree of accuracy. In certain embodiments, the sensor data accessible by the application may include sensor data generated from gyroscopes, accelerometers, magnetometers, eye trackers, and any other sensors located within the virtual reality system. As an example and not by way of limitation, the sensors can be located on the virtual reality headset and the virtual reality controller. In certain embodiments, the virtual reality system may receive a request from a third-party application to access sensor data. The virtual reality system may process the request and determine whether the third-party application requesting the sensor data is currently connected to the user. As an example and not by way of limitation, the virtual reality system may determine whether the user is actively interacting with the application, such as interacting with the keyboard of the application, scrolling through a web page of a browser application, or hovering over an element of the application (e.g., a pointer on the application). In certain embodiments, the virtual reality system may authorize the third-party application to which the user is currently connected to receive the sensor data. Conversely, in certain embodiments, if the user is not currently connected to the application (e.g., an application running in the background), the virtual reality system may prevent the third-party application from receiving the sensor data. After making the determination, the virtual reality system may send the sensor data to the authorized application to receive the sensor data. In certain embodiments, the user may authorize a specific third-party application to receive the sensor data.

[0006] Another object of the disclosed method is to improve generation of user interface elements (e.g., keyboards, radial menus, etc.) in a virtual reality environment. In a particular embodiment, a virtual reality system may provide a method for generating user interface elements for an application within a virtual reality environment using location data. Within the virtual reality environment, a user may view various content. The generation of the user interface element may obscure a portion of the application that is important for the user to view. As an example and not by way of limitation, the obscured portion may include content, such as a question that the user answers with the generated user interface element. As another example and not by way of limitation, the obscured content may include a predicted input to a search field box.

[0007] To avoid obstructing a portion of an application within the virtual reality environment, the user may be allowed to manually move the user interface element to a different location. As an example and not by way of limitation, the user may click on a keyboard and drag and drop the keyboard to another location. As another example and not by way of limitation, the user may use gestures to move the keyboard from one location to another. The virtual reality system may store position data associated with the user interface element to identify locations within the virtual reality environment that may not obstruct portions of the application that the user wants to view. Assume that the user may move the user interface element out of the way to view any content that the user wants to see connected to the user interface element. The virtual reality system may use the stored data to generate and display the user interface element at a location that does not obstruct any portion of the application that the user wants to view. In certain embodiments, the virtual reality system may store and compile position data from multiple users to accurately identify areas within the display area of ​​the application that may contain content.

[0008] Another objective of the disclosed method is to provide copy and paste functionality within a virtual reality environment. Users may take for granted that there are many features when using their devices (e.g., smartphones, tablets, laptops, etc.) to browse on the Internet, social media, etc. One of these features may include a copy and paste function. However, due to the differences between virtual reality environments and two-dimensional (2D) screens such as phone screens, current virtual reality environments may not support the copy and paste functions that users expect. In addition, although most devices may have a cursor, virtual reality environments are different because users have one or two virtual reality input devices (e.g., hand remote controls) that interact with three-dimensional (3D) space. This can cause problems because the movement of the remote control in a 3D VR environment does not translate to the same motion accuracy as a cursor on a 2D plane. For example, using a smartphone with a 2D screen to perform a copy and paste function is relatively straightforward, with just a tap and hold to highlight text. The user can select text by moving his or her finger along the screen, giving them fine control over what is selected. This process is even more simplified for desktop and laptop computers with cursors that indicate where the text highlighting begins and ends. However, in a VR environment, a user is provided with a VR input device (e.g., a hand remote control) that projects a pointer onto surfaces in the VR environment. In a VR environment, these surfaces may be far away from the user. Thus, when a user projects a pointer and attempts to select text that they wish to copy and paste in a manner similar to a system with a 2D screen, the process is difficult for the user, especially if the text is small and far away (because when projected, any deviation in movement is magnified in proportion to the distance). Additionally, gravity and hand tremors may add to the difficulty and imperfect manner in which text is highlighted, because the level of motor control required to handle a pointer is much higher than a cursor on a 2D screen.

[0009] To correct this problem, in certain embodiments, the virtual reality system may receive position data of a pointer of a virtual reality input device projected onto a surface within the virtual reality environment (e.g., a panel displaying an application). The virtual reality system may determine a path in the virtual reality environment from the position data projected onto the surface. In certain embodiments, the virtual reality system may determine the path of the pointer within a predetermined period of time. For example, the virtual reality system may determine the path of the pointer within the past 3 seconds. After determining the path, the client system may identify one or more words on the surface displayed in the virtual reality environment that are surrounded by the path. The virtual reality system may receive an instruction from the user to copy the one or more words surrounded by the path. In certain embodiments, the one or more words may be stored in a temporary memory. The virtual reality system may receive input from the user indicating a location in the virtual reality environment. As an example and not by way of limitation, the user may click on a message box of the application. After receiving the input indicating a location in the virtual reality environment, the virtual reality system may receive another instruction from the user to paste the one or more words. In certain embodiments, the user may select a paste option to perform a paste function. The virtual reality system may perform the paste function and display the one or more words at the location indicated by the user.

[0010] In certain embodiments, a user may wish to resize a text box associated with one or more words enclosed by a path. For example, the user may have performed a large swipe gesture and enclosed a larger portion of content than desired. The virtual reality system may receive an instruction to resize the text box. After initiating the resizing process, the user may use one or two virtual reality input devices (e.g., a hand remote control) to resize the text box. As an example and not by way of limitation, the user may use the pointers of two virtual reality input devices to point to the corners of the text box and position the corners around the desired content. As another example and not by way of limitation, the user may use the pointers of two virtual reality input devices and select the left and right sides of the text box. In certain embodiments, the resizing process may end after releasing a button of the virtual reality input device. In certain embodiments, the resizing process may end after the virtual reality system receives an input to end the resizing process.

[0011] Another goal of the disclosed method is to implement a redirection mode in a virtual reality environment. Typically, users use their devices in a variety of directions. For example, users can use their smartphones and other devices while sitting at a dining table, walking on the street, sitting in a car, lying in bed, or even in the shower. However, current implementations of virtual reality systems are locked in one direction (e.g., when a user moves in a virtual cinema, the virtual cinema appears to be fixed in space) or in a head-locked mode (e.g., the entire cinema (including the screen and seats) will always appear as if the user is facing forward in the cinema, even if the user's body is facing up). Therefore, if the virtual reality cinema is fixed in space, the user will have to face forward to view the screen. If the virtual reality cinema is fixed relative to the user's head, the user will have an unrealistic experience of seeing the seat in front of him as if he is sitting down, even if he may be lying down. The separation between the visual scene and the user's body position may cause the user to feel confused and nauseous. To add a more flexible environment, specific elements in the virtual reality environment can be redirected to better accommodate the user. For example, a virtual movie screen can be separated from a virtual theater and allow for custom anchoring to a different location / orientation that is more convenient for the user (e.g., the user may wish to lie down and place the virtual movie screen on the ceiling of the virtual theater). Additionally, a user interface surrounding the user in a virtual reality environment (e.g., a panel containing web pages and other open applications) can be anchored to a specific orientation in the virtual reality environment.

[0012] To implement the redirection mode, certain embodiments of the virtual reality system may first generate a virtual reality panel (e.g., a panel containing a web page, etc.) to be redirected. The virtual reality panel may be fixed relative to a location in the virtual reality environment. As an example and not by way of limitation, the virtual reality panel (e.g., a web page) may be coupled to the center of a virtual reality theater. The virtual reality system may receive input to enable the redirection mode of the virtual reality panel in the virtual reality environment. As an example and not by way of limitation, the virtual reality system may receive a click on a button on a virtual reality input device (e.g., a hand remote control) to enable the redirection mode. In another example and not by way of limitation, the virtual reality system may receive a click on a virtual reality button in the virtual reality environment. Enabling the redirection mode may allow the virtual reality panel to be redirected relative to the user's viewpoint. The virtual reality system may receive sensor data indicating a change in the user's viewpoint. As an example and not by way of limitation, the sensor data may come from one or more sensors located on the virtual reality system (e.g., an accelerometer, a gyroscope, a magnetometer, an eye tracking sensor). The virtual reality system may redirect the virtual reality panel based on the received sensor data. After the user completes the redirection of him or herself (such as lying in bed), the virtual reality system may receive input to disable the redirection mode. As an example and not by way of limitation, the user may disable redirection mode by clicking a button on the manual remote control or by clicking a virtual reality button. Disabling redirection mode may fix the virtual reality panel relative to a new position in the virtual reality environment.

[0013] Another objective of the disclosed method is to implement a redirection mode in a virtual reality environment while driving. For example, a user may want to use a virtual reality system in a driving vehicle. Sensors (e.g., accelerometers, magnetometers, gyroscopes, etc.) may detect the movement of the vehicle and incorrectly adjust the user interface (e.g., virtual reality panel) and / or the virtual reality environment due to any movement (such as a car turning). To address this issue, a redirection mode or "driving mode" may be used that dynamically changes the direction of the user interface based on detected movement associated with external forces (e.g., car movement, airplane movement, etc.). The driving mode may be a redirection mode of the virtual reality panel with damping to accommodate slight movements in the vehicle. To implement the redirection mode in the virtual reality environment while driving, the virtual reality system may receive input to enable the driving redirection mode of the virtual reality environment. Similar to how the redirection mode of the virtual reality panel is enabled, the driving redirection mode may be enabled by clicking a button on a virtual reality input device (e.g., a manual remote control) or by clicking a virtual button. The enabling of the driving redirection mode may set an initial direction of the user's viewpoint relative to the virtual reality environment. As an example and not by way of limitation, if the user is facing a virtual reality theater, where the display is located at the center, the initial direction may be set to the display being located at the center of the virtual reality theater. The virtual reality system may receive sensor data indicating a change in direction. Based on the sensor data, the virtual reality system may adjust a viewpoint of the user relative to the virtual reality environment based on the sensor data. The virtual reality system may readjust the viewpoint of the user back to an initial direction relative to the virtual reality environment. As an example and not by way of limitation, the virtual reality system may adjust the viewpoint so that the display is located at the center of the virtual reality theater. In a particular embodiment, the virtual reality system may determine that the sensor data indicates a change in direction due to movement of a vehicle that the user is occupying, and in response to determining that the change in direction is due to vehicle movement, readjust the viewpoint of the user back to the initial direction relative to the virtual reality environment. As an example and not by way of limitation, if the virtual reality system determines that the viewpoint changed due to the vehicle turning, the viewpoint will return to the initial direction.

[0014] Disclosed herein are a number of different ways to render and interact with a virtual (or augmented) reality environment. A virtual reality system can present a virtual environment that can include a virtual space rendered for display to one or more users. Users can view and interact with this virtual space and the broader virtual environment in any suitable manner. One goal of the disclosed methods is to provide an intuitive experience for users, i.e., to give users a sense of "presence" or the feeling that they are actually in the virtual environment.

[0015] Embodiments of the present invention may include or be implemented in conjunction with an artificial reality system. Artificial reality is a form of reality that has been adjusted in some way before being presented to a user, which may include, for example, virtual reality (VR), augmented reality (AR), mixed reality (MR), hybrid reality, or some combination and / or derivative thereof. Artificial reality content may include fully generated content or generated content combined with captured content (e.g., photos of the real world). Artificial reality content may include video, audio, tactile feedback, or some combination thereof, and any of them may be presented in a single channel or multiple channels (such as stereoscopic video that produces a three-dimensional effect to the audience). In addition, in some embodiments, artificial reality may be associated with, for example, applications, products, accessories, services, or some combination thereof for creating content in artificial reality and / or for (e.g., performing activities in artificial reality) artificial reality. An artificial reality system that provides artificial reality content can be implemented on a variety of platforms, including a head-mounted display (HMD) connected to a host computer system, an independent HMD, a mobile device or computing system, or any other hardware platform capable of providing artificial reality content to one or more viewers.

[0016] The embodiments disclosed herein are merely examples, and the scope of the present disclosure is not limited thereto. Specific embodiments may include all, some, or no components, elements, features, functions, operations, or steps of the embodiments disclosed above. In particular, embodiments according to the present invention are disclosed in the attached claims, and relate to a method, a storage medium, a system, and a computer program product, wherein any feature mentioned in one claim category (e.g., a method) may be claimed in another claim category (e.g., a system). Dependencies or references in the attached claims are selected only for formal reasons. However, any subject matter generated by intentional reference from any previous claim (especially multiple dependencies) may be claimed to disclose and any combination of claims and their features may be claimed, regardless of the dependencies selected in the attached claims. The subject matter that may be claimed includes not only the combination of features set forth in the attached claims, but also any other combination of features in the claims, wherein each feature mentioned in the claims may be combined with any other feature or combination of other features in the claims. In addition, any embodiment and feature described or depicted herein may be claimed in a separate claim and / or in any combination with any embodiment or feature described or depicted herein or with any feature of the attached claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 An example network environment associated with a virtual reality system is shown;

[0018] Figures 2A to 2GAn example of a user interacting with multiple user interface elements in a virtual reality environment is shown;

[0019] FIG. 3A to FIG. 3C An example of a user interacting with multiple applications in a virtual reality environment is shown;

[0020] Figure 4 An example method for distinguishing generated OS content from generated third-party content is shown;

[0021] Figure 5 An example method for sending sensor data to an authorized application is shown;

[0022] FIG. 6A to FIG. 6C An example of a user interacting with a user interface element of an application in a virtual reality environment is shown;

[0023] FIG. 7A to FIG. 7B Another example of a user interacting with a user interface element of an application in a virtual reality environment is shown;

[0024] Figure 8 An example method for generating a user interface element that avoids obstructing a portion of an application in a virtual reality environment is shown;

[0025] 9A to 9H An example of a user copying and pasting content in a virtual reality environment is shown;

[0026] Fig.10 An example method for copying and pasting content in a virtual reality environment is shown;

[0027] FIG. 11A to FIG. 11C An example of a user using a redirection mode in a virtual reality environment is shown;

[0028] FIG. 12A to FIG. 12E An example of a user using a driving redirection mode in a virtual reality environment is shown;

[0029] Fig.13 An example method for utilizing a redirection mode in a virtual reality environment is shown;

[0030] Fig.14 An example computer system is shown. DETAILED DESCRIPTION

[0031] Figure 1 An example network environment 100 associated with a virtual reality system is shown. The network environment 100 includes a user 101 interacting with a client system 130, a social networking system 160, and a third-party system 170 connected to each other via a network 110. Figure 1A particular arrangement of user 101, client system 130, social networking system 160, third-party system 170, and network 110 is shown, but this disclosure contemplates any suitable arrangement of user 101, client system 130, social networking system 160, third-party system 170, and network 110. As one example, and not by way of limitation, two or more of user 101, client system 130, social networking system 160, and third-party system 170 may be directly connected to each other, bypassing network 110. As another example, two or more of client system 130, social networking system 160, and third-party system 170 may be physically or logically co-located, in whole or in part, with each other. Furthermore, although Figure 1 A particular number of users 101, client systems 130, social networking systems 160, third-party systems 170, and networks 110 are shown, but the present disclosure contemplates any suitable number of client systems 130, social networking systems 160, third-party systems 170, and networks 110. As an example, and not by way of limitation, network environment 100 may include a plurality of users 101, client systems 130, social networking systems 160, third-party systems 170, and networks 110.

[0032] The present disclosure contemplates any suitable network 110. As an example, and not by way of limitation, one or more portions of network 110 may include an ad hoc network, an intranet, an extranet, a virtual private network (VPN), a local area network (LAN), a wireless LAN (WLAN), a wide area network (WAN), a wireless WAN (WWAN), a metropolitan area network (MAN), a portion of the Internet, a portion of a public switched telephone network (PSTN), a cellular telephone network, or a combination of two or more thereof. Network 110 may include one or more networks 110.

[0033] The links 150 may connect the client systems 130, the social networking systems 160, and the third-party systems 170 to the communication network 110 or to each other. The present disclosure contemplates any suitable links 150. In a particular embodiment, one or more links 150 include one or more wired (such as, for example, digital subscriber line (DSL) or cable data service interface specification (DOCSIS)), wireless (such as, for example, Wi-Fi or Worldwide Interoperability for Microwave Access (WiMAX)), or optical (such as, for example, synchronous optical network (SONET) or synchronous digital hierarchy (SDH)) links. In a particular embodiment, one or more links 150 each include a self-organizing network, an intranet, an extranet, a VPN, a LAN, a WLAN, a WAN, a WWAN, a MAN, a portion of the Internet, a portion of the PSTN, a network based on cellular technology, a network based on satellite communication technology, another link 150, or a combination of two or more such links 150. The links 150 do not have to be the same throughout the network device 100. One or more first links 150 may differ from one or more second links 150 in one or more aspects.

[0034] In particular embodiments, client system 130 may be an electronic device that includes hardware, software, or embedded logic elements or a combination of two or more such elements and is capable of performing appropriate functions implemented or supported by client system 130. By way of example, and not by way of limitation, client system 130 may include a computer system such as a desktop computer, a notebook or laptop computer, a netbook, a tablet computer, an e-reader, a GPS device, a camera, a personal digital assistant (PDA), a handheld electronic device, a cellular phone, a smart phone, a virtual reality headset and controller, other suitable electronic devices, or any suitable combination thereof. The present disclosure contemplates any suitable client system 130. Client system 130 may enable network users on client system 130 to access network 110. Client system 130 may enable its users to communicate with other users on other client systems 130. Client system 130 may generate a virtual reality environment for users to interact with content.

[0035] In certain embodiments, the client system 130 may include a virtual reality (or augmented reality) head mounted device 132 (such as OCULUS RIFT, etc.) and a virtual reality input device 134 (such as a virtual reality controller). A user at the client system 130 may wear the virtual reality head mounted device 132 and use the virtual reality input device to interact with a virtual reality environment 136 generated by the virtual reality head mounted device 132. Although not shown, the client system 130 may also include a separate processing computer and / or any other components of a virtual reality system. The virtual reality head mounted device 132 may generate a virtual reality environment 136, which may include system content 138 (including but not limited to an operating system), such as software or firmware updates, and also include third-party content 140, such as content from an application or content dynamically downloaded from the Internet (e.g., web content). The virtual reality head mounted device 132 may include sensors 142, such as accelerometers, gyroscopes, magnetometers, to generate sensor data that tracks the position of the head mounted device 132. The head mounted device 132 may also include an eye tracker to track the position of the user's eyes or their viewing direction. The client system 130 may use data from sensors 142 to determine velocity, direction, and gravity relative to the headset. The virtual reality input device 134 may include sensors 144, such as accelerometers, gyroscopes, magnetometers, and touch sensors, to generate sensor data that tracks the position of the input device 134 and the position of the user's fingers. The client system 130 may utilize outside-in tracking, where a tracking camera (not shown) is placed outside of the virtual reality headset 132 and within the line of sight of the virtual reality headset 132. In outside-in tracking, the tracking camera may track the position of the virtual reality headset 132 (e.g., by tracking one or more infrared LED markers on the virtual reality headset 132). Alternatively or in addition, the client system 130 may utilize inside-out tracking, where a tracking camera (not shown) may be placed on or inside the virtual reality headset 132 itself. In inside-out tracking, the tracking camera may capture images of its surroundings in the real world, and may use the changing perspective of the real world to determine its own position in space.

[0036] The third-party content 140 may include a web browser 132, such as MICROSOFT INTERNET EXPLORER, GOOGLE CHROME, or MOZILLA FIREFOX, and may have one or more additional components, plug-ins, or other extensions, such as TOOLBAR or YAHOO TOOLBAR. A user on the client system 130 may enter a uniform resource locator (URL) or other address to direct the web browser to a specific server (such as server 162 or a server associated with the third-party system 170), and the web browser may generate a hypertext transfer protocol (HTTP) request and transmit the HTTP request to the server. The server may accept the HTTP request and transmit one or more hypertext markup language (HTML) files in response to the HTTP request to the client system 130. The client system 130 may render a network interface (e.g., a web page) based on the HTML file from the server for presentation to the user. The present disclosure contemplates any suitable source file. As an example, and not by way of limitation, the network interface may be rendered from an HTML file, an extensible hypertext markup language (XHTML) file, or an extensible markup language (XML) file according to specific needs. Such an interface may also execute scripts, such as, for example and without limitation, scripts written in JAVASCRIPT, JAVA, MICROSOFT SILVERLIGHT, a combination of markup languages ​​and scripts such as AJAX (Asynchronous JAVASCRIPT and XML), and the like. In this document, where appropriate, references to a web interface include one or more corresponding source files (files that a browser may use to render the web interface), and vice versa.

[0037] In particular embodiments, social networking system 160 may be a network-addressable computing system that may host an online social network. Social networking system 160 may generate, store, receive, and transmit social networking data, such as, for example, user profile data, concept profile data, social graph information, or other suitable data related to an online social network. Social networking system 160 may be accessed by other components of network environment 100 directly or via network 110. As an example and not by way of limitation, client system 130 may access social networking system 160 directly or via network 110 using a web browser of third-party content 140 or a local application associated with social networking system 160 (e.g., a mobile social networking application, a messaging application, another suitable application, or any combination thereof). In particular embodiments, social networking system 160 may include one or more servers 162. Each server 162 may be a unitary server or a distributed server that spans multiple computers or multiple data centers. Server 162 may be of various types, such as, for example and without limitation, a web server, a news server, a mail server, a message server, an advertising server, a file server, an application server, an exchange server, a database server, a proxy server, another server suitable for performing the functions or processes described herein, or any combination thereof. In particular embodiments, each server 162 may include hardware, software, or embedded logic elements or a combination of two or more such elements for performing the appropriate functions implemented or supported by server 162. In particular embodiments, social networking system 160 may include one or more data stores 164. Data stores 164 may be used to store various types of information. In particular embodiments, the information stored in data stores 164 may be organized according to a particular data structure. In particular embodiments, each data store 164 may be an associative, columnar, correlational, or other suitable database. Although the present disclosure describes or illustrates a particular type of database, the present disclosure contemplates any suitable type of database. Particular embodiments may provide an interface that enables client system 130, social networking system 160, or third-party system 170 to manage, retrieve, add, or delete information stored in data stores 164.

[0038] In particular embodiments, social networking system 160 may store one or more social graphs within one or more data stores 164. In particular embodiments, a social graph may include a plurality of nodes - which may include a plurality of user nodes (each corresponding to a particular user) or a plurality of concept nodes (each corresponding to a particular concept) - and a plurality of edges connecting the nodes. Social networking system 160 may provide users of an online social network with the ability to communicate and interact with other users. In particular embodiments, a user may join an online social network via social networking system 160 and then add connections (e.g., relationships) with a plurality of other users of social networking system 160 with whom the user wishes to connect. As used herein, the term "friend" may refer to any other user of social networking system 160 with whom the user has formed a connection, association, or relationship via social networking system 160.

[0039] In particular embodiments, social networking system 160 may provide users with the ability to take actions on various types of items or objects supported by social networking system 160. By way of example and not limitation, items and objects may include groups or social networks to which users of social networking system 160 belong, events or calendar items that may be of interest to the user, computer-based applications that the user may use, transactions that allow users to purchase or sell items through the service, interactions with advertisements that the user may perform, or other suitable items or objects. Users may interact with any object that can be represented within social networking system 160 or by an external system of third-party system 170 that is separate from social networking system 160 and coupled to social networking system 160 via network 110.

[0040] In certain embodiments, social networking system 160 can link various entities. As an example, and not by way of limitation, social networking system 160 can enable users to interact with each other and receive content from third-party systems 170 or other entities, or allow users to interact with these entities through an application programming interface (API) or other communication channels.

[0041] In particular embodiments, third-party systems 170 may include one or more types of servers, one or more data stores, one or more interfaces (including but not limited to APIs), one or more web services, one or more content sources, one or more networks, or any other suitable component that can communicate with a server, for example. Third-party systems 170 may be operated by an entity that is different from the entity that operates social-networking system 160. In particular embodiments, however, social-networking system 160 and third-party systems 170 may operate in conjunction with one another to provide social-networking services to users of social-networking system 160 or third-party systems 170. In this sense, social-networking system 160 may provide a platform or backbone that other systems (such as third-party systems 170) may use to provide social-networking services and functionality to users over the Internet.

[0042] In particular embodiments, third-party system 170 may include a third-party content object provider. A third-party content object provider may include one or more sources of content objects that may be delivered to client system 130. As an example, and not by way of limitation, a content object may include information about things or activities of interest to a user, such as, for example, movie showtimes, movie reviews, restaurant reviews, restaurant menus, product information and reviews, or other suitable information. As another example, and not by way of limitation, a content object may include an incentive content object, such as a coupon, discount ticket, gift certificate, or other suitable incentive object.

[0043] In particular embodiments, social networking system 160 also includes user-generated content objects that can enhance a user's interaction with social networking system 160. User-generated content can include anything that a user can add, upload, send, or "post" to social networking system 160. As an example, and not by way of limitation, a user transmits a post to social networking system 160 from client system 130. Posts can include data such as status updates or other textual data, location information, photos, videos, links, music, or other similar data or media. Posts can also be added to social networking system 160 by third-party systems 170 through "communication channels" such as news feeds or streams.

[0044] In certain embodiments, the social networking system 160 may include various servers, subsystems, programs, modules, records, and data stores. In certain embodiments, the social networking system 160 may include one or more of the following: a network server, an activity recorder, an API request server, a relevance and ranking engine, a content object classifier, a notification controller, an activity record, a third-party content object exposure record, an inference module, an authorization / privacy server, a search module, an advertising targeting module, a user interface module, a user profile store, a connection store, a third-party content store, or a location store. The social networking system 160 may also include suitable components, such as a network interface, a security mechanism, a load balancer, a failover server, an administration and network operations console, other suitable components, or any suitable combination thereof. In certain embodiments, the social networking system 160 may include one or more user profile stores for storing user profiles. User profiles may include, for example, biographical information, demographic information, behavioral information, social information, or other types of descriptive information, such as work experience, education, hobbies or preferences, interests, affinities, or locations. Interest information may include interests associated with one or more categories. Categories may be general or specific categories. As an example, and not by way of limitation, if a user "likes" merchandise about a certain brand of shoes, the category may be "shoes" or "clothing" for that brand or a general category. A connection store may be used to store connection information about a user. The connection information may represent users who have similar or common work experience, group affiliations, hobbies, education, or are related in any way or share common attributes. The connection information may also include user-defined connections between different users and content (internal and external). A network server may be used to link the social networking system 160 and one or more client systems 130 or one or more third-party systems 170 via the network 110. The network server may include a mail server or other messaging functionality for receiving and routing messages between the social networking system 160 and one or more client systems 130. The API request server may allow a third-party system 170 to access information from the social networking system 160 by calling one or more APIs. An activity logger may be used to receive communications from a network server about a user's activities on or away from the social networking system 160. In conjunction with the activity log, a third-party content object log may maintain a user's exposure to a third-party content object. The notification controller may provide information about the content object to the client system 130. The information may be pushed to the client system 130 as a notification, or may be pulled from the client system 130 in response to a request received from the client system 130. The authorization server may be used to enforce one or more privacy settings of users of the social networking system 160.The user's privacy settings determine how certain information associated with the user may be shared. The authorization server may allow the user to opt in or out of activities recorded by the social networking system 160 or shared with other systems (e.g., third-party systems 170), such as, for example, by setting appropriate privacy settings. A third-party content object store may be used to store content objects received from third parties (such as third-party systems 170). A location store may be used to store location information received from a client system 130 associated with a user. The advertising pricing module may incorporate social information, current time, location information, or other suitable information to provide relevant advertisements to the user in the form of notifications.

[0045] Figures 2A to 2G An example process for maintaining user safety while using user interface elements in a virtual reality environment 200 is shown. In certain embodiments, a client system 130 or a virtual reality system may render a virtual space for display to a user on a display device. In certain embodiments, the virtual reality system may be a local system including a device that is rendered locally relative to a user of the virtual reality system. In certain embodiments, the virtual reality system may be a remote device (e.g., a remote server computing machine), or may at least include a remote device. As an example and not by way of limitation, a virtual reality system may be defined as including a server of a social networking system 160. As another example and not by way of limitation, a virtual reality system may be defined as including a server of a social networking system 160 and a local computing device. In certain embodiments, the virtual space may be an augmented reality space in which virtual elements are overlaid on the real world. As an example and not by way of limitation, the virtual reality system may continuously capture images of the real world (e.g., using a camera on a user's head-mounted device) and overlay virtual objects or avatars of other users on these images so that the user can interact with the real world and the virtual world simultaneously. In certain embodiments, the user may view the virtual space using a head-mounted device. As an example and not by way of limitation, reference is made to Figure 2A , a user may mount a virtual reality headset 132 on the user's head. In certain embodiments, the headset apparatus may be an apparatus that may be mounted, placed, or otherwise connected to the user's head. In certain embodiments, the headset apparatus may include a display mechanism that displays the virtual space area to the user. As an example and not by way of limitation, the display mechanism may include a screen that displays the virtual space area. As another example and not by way of limitation, the display mechanism may be a projector that projects the display of the virtual space area directly into a sweet spot in the user's eyes (e.g., the fovea of ​​each of the user's eyes).

[0046] In certain embodiments, the client system 130 may use the virtual reality headset 132 to render a panel 202 in a virtual space that includes one or more applications 210a-210h, such as third-party applications including games, web browsers, and any other type of application that the virtual reality system may support. Figure 2A The user 101 is shown wearing a virtual reality headset 132 and using a virtual reality input device 134 to interact with the virtual reality environment 200. In certain embodiments, the user 101 can open the panel 202 by interacting with the virtual reality environment 200 (e.g., touching a virtual reality element within the virtual reality environment 200) or by inputting an input to the virtual reality input device 134 (e.g., clicking a button). Figure 2A As shown, the user 101 can see the pointer 212 and the pointer path 214 within the virtual reality environment 200 to visualize the location that the user 101 is pointing to with the virtual reality input device 134. The user 101 can also select a desired location by pointing the pointer 212 to the desired location and inputting input to the virtual reality input device 134 (e.g., clicking a button). Figure 2A Any of the applications 210a-210h shown (eg, application 210h, which may be a web browser).

[0047] Figure 2B The result of selecting application 210h (e.g., a web browser) is shown, which may result in displaying panel 204 of application 210h. In certain embodiments, application 210h may include a plurality of selectable media 216a-216c and media 216c is used for logging in. Figure 2B As shown, the user 101 may select the medium 216c by pointing the pointer 212 to the desired location and inputting into the virtual reality input device 134. In certain embodiments, the application 210h may be grayed out to highlight that it is an open application. Figure 2C It is shown that after selecting the medium 216c, a login box 218 related to the third-party content may appear, which may include input fields 220a-220b. Figure 2C As shown, user 101 can "click" on input field 220a to continue entering information to log in to use the third-party content associated with media 216c. Although login box 218 is associated with media 216c, login box 218 can be associated with other third-party content, such as other applications or other media 216a-216b. Figure 2DIt is shown that a dedicated plane 222 is generated to display a user interface element 224 (e.g., a keyboard) to receive input from the user 101. In a particular embodiment, the dedicated plane 222 can be a storage space for generated third-party content. As an example and not by way of limitation, the dedicated plane 222 can be used for a third-party keyboard, which can be different from the system keyboard to warn the user to be careful whether to use the keyboard and / or whether the user should stop interacting with the third-party application that generates the keyboard. In a particular embodiment, the dedicated plane 222 is transparent and is only shown for reference to the location where the third-party user interface element (such as the user interface element 224) is displayed after being generated by the client system 130. The dedicated plane 222 can represent a plane that can be rotated 360 degrees around the user 101. In a particular embodiment, the user interface element 224 can be embodied as any other suitable user interface element that receives user input. In a particular embodiment, the client system 130 can also generate a warning (e.g., a pop-up box with a warning) in the dedicated plane 222 to indicate that the user interface element 224 is not a system keyboard, and can also warn the user not to use the keyboard. To this end, client system 130 may determine that user interface element 224 is a third-party keyboard and issue an alert in response to this determination.

[0048] Figure 2E 2 shows system content (e.g., a user interface generated by an operating system) appearing in the virtual reality environment 200 in the form of a system update box 226 including a plurality of confirmation buttons 228a-228b. Figure 2E As further shown, user 101 can select a "yes" confirmation button 228a to confirm the update to the system by entering input into virtual reality input device 134. In particular embodiments, OS content can be distinguished from third-party content by including additional elements (e.g., a border around the OS content), having a different style or layout (e.g., a black keyboard versus a light keyboard), or any combination thereof. In particular embodiments, client system 130 can determine whether the request for a keyboard is for a system keyboard or a third-party keyboard. As an example and not by way of limitation, client system 130 can determine whether the system generates a keyboard to be displayed for user input or whether a third-party application generates a keyboard for user input. Figure 2F The result of selecting the “Yes” confirmation button 228a is shown, which may result in displaying input fields 230a-230b, which the user 101 may select by pointing the pointer 212 to a desired location and inputting input into the virtual reality input device 134 (e.g., clicking a button). Figure 2GIt is shown that after selecting input field 230a, another dedicated plane 232 is generated with user interface element 234 (e.g., keyboard) for user 101 to enter their information into input field 230a. In certain embodiments, dedicated plane 222 and dedicated plane 232 are separate planes, referencing the space where generated user interface elements can be displayed for user 101 to interact with. In certain embodiments, dedicated plane 222 can be a plane dedicated to third-party content (e.g., pop-up windows, content, tools, user interfaces, etc.), such as from applications, etc. In certain embodiments, dedicated plane 232 can be a plane dedicated to OS content, such as a user interface generated by the system, etc. In certain embodiments, when a third-party application requests the OS to provide a user interface or content generated by the OS (e.g., a third-party application can call an OS API to call the system's virtual keyboard), the resulting interface or content can be displayed in the dedicated plane 232 of the OS. In contrast, if a third-party application generates its own content or user interface, the generated content or user interface can be presented in dedicated plane 222. In certain embodiments, dedicated plane 222 may be different from dedicated plane 232 based on a variety of factors such as depth, layout, style, etc. In certain embodiments, dedicated plane 232 may be closer to user 101 in virtual reality environment 200 than dedicated plane 222 to distinguish dedicated plane 232 from dedicated plane 222. By making dedicated plane 232 closer, user 101 is able to recognize that user interface elements 234 in dedicated plane 232 are OS content. In certain embodiments, by distinguishing between plane 222 dedicated to third-party content and plane 232 dedicated to OS content, the security of user 101 may be improved by preventing third-party entities from generating third-party content that can be used to attempt to obtain confidential information of user 101.

[0049] FIG. 3A to FIG. 3C 1 shows an example multitasking situation in which user 101 may participate in virtual reality environment 300. Figures 2A to 2G , the client system 130 can use the virtual reality headset 132 to render a virtual reality environment including the virtual reality elements in question. Figure 3A A virtual reality environment 300 is shown including sensor data 302 collected by sensors 142 , 144 in the background and panels 304 , 306 including an application that the user 101 has selected. Figure 3A The user 101 is shown pointing the virtual reality input device 134 to a launch button 308 of an application (e.g., a VR game) using a pointer 312 and a pointer path 314. In certain embodiments, the panels 304 and 306 can display any type of selected application, as described above. In certain embodiments, the panel 306 can include an interactive field 310 (e.g., a search box). Figure 3AAs shown, the user 101 may “click” the launch button 308 of the application by pointing the pointer 312 to the desired location and inputting input into the virtual reality input device 134 (e.g., clicking the button).

[0050] Figure 3B The result of pressing the launch button 308 is shown, which can cause the virtual reality environment 300 to change according to the content related to the application (e.g., a VR game) loaded in the panel 304. In certain embodiments, the virtual reality environment 300 does not have to change and can remain unchanged. In certain embodiments, the selection of an application of the panel 304 can authorize the application of the panel 304 to receive sensor data 302 from the sensors 142, 144. In certain embodiments, the application of the panel 304 can request that the sensor data 302 be sent to the application of the panel 304, and the client system 130 can identify that the application is authorized to receive the sensor data 302 based on the user 101's recent connection with the application (e.g., selecting the application to launch the application). In certain embodiments, the sensor data 302 can be used to interact with the virtual reality environment 300 in the context of the application, such as moving within the virtual reality environment 300 and viewing the content of the virtual reality environment 300. In certain embodiments, the sensor data 302 can be associated with a particular application, and such information can be used by the system to determine whether a particular application (e.g., the application of the panel 304) can access the particular sensor data 302. As shown in FIG. Figure 3B As shown, the user 101 can select an interaction box 310 of an application displayed in the panel 306, and interact with the virtual reality environment 300 generated by the application of the panel 304 by pointing a pointer 312 to a desired location and inputting input to the virtual reality input device 134 (e.g., clicking a button). When the user's interaction is directed to the virtual reality environment 300 generated by the application of the panel 304, the measured sensor data 302 (e.g., eye tracking data, position and orientation data associated with a head mounted device, movement of a controller, etc.) is available to the application of the panel 304. However, when the user 101 interacts with the application displayed in the panel 306 (e.g., including looking at the panel 306, pointing to the panel 306, or clicking within the panel 306), the application of the panel 304 may restrict access to the corresponding sensor data.

[0051] Figure 3CThe result of selecting the interaction box 310 is shown, which can result in the generation of a dedicated plane 316 and the display of a user interface element 318 to receive input to the interaction box 310. In certain embodiments, the interaction with the application of the panel 306 can revoke the authorization of the application of the panel 304 to receive the sensor data 302, and authorize the application of the panel 306 to receive the sensor data 302. In certain embodiments, the user's interaction or connection activity with the application can be active, such as clicking a button to send a command to the target application, and / or passive, such as looking at the target application or gesturing to the target application. In certain embodiments, the interaction with the application of the panel 306 can only authorize the application of the panel 306 to receive data, while maintaining the authorization of the application of the panel 304 to receive the sensor data 302. In certain embodiments, the client system 130 can identify the purpose of the user interface element 318 and can revoke the access to the sensor data 302 for the application that did not request the user interface element 318. As an example and not by way of limitation, if the user interface element 318 (e.g., keyboard) is used to enter a search query, then other applications may retain their authorization to access the sensor data 302, but if it is used to enter the credentials of the user 101, then other applications may revoke their authorization to access the sensor data 302. In certain embodiments, the client system 130 may bind the sensor data 302 to a currently active application, which may be defined as the application to which the user 101 is currently connected. In certain embodiments, the client system 130 may determine that the user interface element 318 has been generated and allow the sensor data 302 to be sent to the application associated with the user interface element 318. By stopping the sending of sensor data 302 to applications of the panel 304, the client system 130 may increase the security of the user 101 by unnecessarily sending sensor data 302 to applications to which the user 101 is not currently connected. For example, if sensor data 302 continues to be sent to any open application, an application with malicious intent may determine that client system 130 has activated user interface element 318 that requires confidential information from user 101, and monitor sensor data 302, and infer user 101 input fairly accurately from sensor data 302. Continuing with this example, the malicious application may access eye tracking data and motion data (e.g., from the controller and headset) to narrow down the possibilities of what the user may enter into user interface element 318, and easily guess confidential information (e.g., username, password, birthday, social security number, etc.) entered into user interface element 318. In certain embodiments, client system 130 may stop sending only a subset of sensor data 302 to any open application. As an example and not by way of limitation, client system 130 may continue to send location data and revoke the application's authorization to receive eye tracking / headset orientation / hand motion data, which may be used to potentially compromise the safety of user 101.In certain embodiments, user 101 may explicitly permit an application to continue receiving sensor data 302 from client system 130 even when the user is interacting with another application or another application has become the active application. A list of authorized applications to receive sensor data 302 may be generated for user 101 to maintain a catalog of permitted applications. In certain embodiments, the list of applications may be added to a whitelist or blacklist to automatically determine whether a particular application is authorized to receive sensor data 302.

[0052] Figure 4 An example method 400 for distinguishing between generated OS content and generated third-party content and selectively displaying the content in different display planes accordingly is shown. The method may begin at step 410, where a client system (e.g., a virtual reality system) may receive a request to display a user interface element in a virtual reality environment. For example, the request may come from an application (e.g., including a web page interpreted by an application such as a web browser). At step 420, the client system may determine whether the requested user interface element is a system user interface (e.g., a keyboard generated by the OS) or a third-party user interface (e.g., a keyboard generated by a third-party application, rather than a keyboard generated by the OS in response to a request / call from the third-party application). In a particular embodiment, the OS may generally generate user interface elements for use by third-party applications, and only in certain cases (e.g., a third-party application with malicious intent) will a third-party user interface element be generated to receive user input. At step 430, the client system may perform a determination. If the client system determines that the requested user interface is a system user interface, the method may proceed to step 440, where the client system may generate a system OS user interface element for a first dedicated plane of the virtual reality environment. If the client system determines that the requested user interface element is not a system user interface element, the method can proceed to step 450, where the client system can generate a third-party application user interface element for the second dedicated plane of the virtual reality environment. In certain embodiments, the first dedicated plane can be different from the second dedicated plane. Certain embodiments can repeat the steps described above where appropriate. Figure 4 Although the present disclosure will Figure 4 Certain steps of the method are described and shown as occurring in a particular order, but the present disclosure contemplates occurring in any suitable order. Figure 4 Furthermore, although this disclosure describes and illustrates example methods (including Figure 4 ), but the present disclosure contemplates any suitable method (including any suitable steps) for distinguishing generated OS content from generated third-party content, which steps may include, where appropriate, Figure 4 In addition, although the present disclosure describes and illustrates the implementation of Figure 4 specific components, devices or systems for specific steps of the method, but the present disclosure contemplates performing Figure 4 Any suitable combination of any suitable components, devices or systems of any suitable steps of the method.

[0053] Figure 5 An example method 500 for sending sensor data to an authorized application is shown. The method may begin at step 510, where a client system (e.g., a virtual reality system) may receive a request to access sensor data from a third-party application. For example, a VR game may request access to sensor data from a client system so that a user can operate in a virtual reality environment generated by the VR game. At step 520, the client system may determine whether the third-party application is a currently active application. In a particular embodiment, the currently active application may be an application that a user associated with the client system is currently connected to. As an example and not by way of limitation, the client system may identify the application that the user last interacted with, such as selecting the application, hovering a pointer over the application, etc. At step 530, the client system may determine whether the user is currently interacting with a third-party application. If it is determined that the user is currently connected to an application requesting sensor data, then at step 540, the client system may authorize the third-party application to receive sensor data from the client system. If it is determined that the user is not connected to an application requesting sensor data, then at step 550, the client system may prevent the third-party application from receiving sensor data from the client system. At step 560, the client system may send sensor data to a third-party application that has been authorized to receive data. Where appropriate, specific examples may be repeated Figure 5 Although the present disclosure will Figure 5 Certain steps of the method are described and shown as occurring in a particular order, but the present disclosure contemplates occurring in any suitable order. Figure 5 In addition, although the present disclosure describes and illustrates an example method for sending sensor data to an authorized application (including Figure 5 ), but the present disclosure contemplates any suitable method (including any suitable steps) for sending sensor data to an authorized application, which steps may include, where appropriate: Figure 5 In addition, although the present disclosure describes and illustrates the implementation of Figure 5 specific components, devices or systems for specific steps of the method, but the present disclosure contemplates performing Figure 5 Any suitable combination of any suitable components, devices or systems of any suitable steps of the method.

[0054] FIG. 6A to FIG. 6C An example process for maintaining a user interface position database while user 101 is participating in virtual reality environment 600 is shown. Fig. 6A The user 101 is shown wearing a virtual reality headset 132 and using a virtual reality input device 134 to interact with the virtual reality environment 600. Figures 2A to 2G , the client system 130 can use the virtual reality head mounted device 132 to render a virtual reality environment including the above-mentioned virtual reality elements. Since the current implementation of user interface elements (e.g., keyboards) can block content when receiving input from a user, it is necessary to better position the position of the user interface elements to avoid blocking the content and / or part of the display. This is because the content may be related to or interesting to the user's input, such as questions answered by the user on a form. In addition, many applications can provide functions when the user provides input on the user interface element. The function can be a predictive input, such as some kind of auto-complete function. However, in the current implementation of this function, the predictive input can usually be located below the input box. Typically, the user interface element can also be located below the input box and will therefore block the predictive input. In this way, the user interface element needs to be better positioned to allow the user to see the content of the page and / or other content that can be displayed to the user.

[0055] In particular embodiments, virtual reality environment 600 may include panel 602 that displays an application selected by user 101 . Fig. 6A The user 101 is shown pointing the virtual reality input device 134 to an interactive field 604 (e.g., a search box) of an application (e.g., Facebook) using a pointer 606 and a pointer path 608. In certain embodiments, the panel 602 may display any type of application as described above. Fig. 6A As shown, user 101 can "click" the interactive field 604 of the application by pointing pointer 606 to the desired location and inputting input into virtual reality input device 134 (e.g., clicking a button). This can be interpreted as a request to access the user interface element associated with the interactive field box 604.

[0056] Figure 6BThe result of selecting the interactive field 604 is shown, which can result in the generation of a dedicated plane 610 and the display of a search field box 612 with a user interface element 614 (e.g., a keyboard). In certain embodiments, the search field box 612 may include multiple predicted inputs 616. The predicted input 616 may be an input to complete the search box 612 that the user 101 can select. As an example and not by way of limitation, the predicted input 616 may be a recent search conducted in the search box 612. In certain embodiments, the predicted input 616 may be selected to execute the query. Although the dedicated plane 610 is depicted as being separate from the panel 602, in certain embodiments, the dedicated plane 610 with its generated content (e.g., the search field box 612 and the user interface element 614) may be embedded in the panel 602 as content of the application of the panel 602. When the user interface element 614 is generated and displayed, the user interface element may obscure a portion of the application displayed in the panel 602. As an example and not by way of limitation, the user interface element 614 may obscure the predicted input 616. As shown in FIG. Figure 6B As shown, user 101 can select user interface element 614 at pointer position 606a and move user interface element 614 to pointer position 606b along path 618. Virtual reality environment 600 may include position data 620 associated with user interface element 614 and store position data 620 in keyboard position database 622. In certain embodiments, position data 620 may be associated with other elements of an application of panel 602 (e.g., search box 612), and keyboard position database 622 may be a general database that stores position data 620 associated with other elements.

[0057] Figure 6CThe result of moving the user interface element 614 to the pointer position 606b is shown. In a particular embodiment, the client system 130 may store location data 620 associated with the location of the user interface element 614. The location data 620 may be associated with a particular application. As an example and not by way of limitation, an application (e.g., Facebook) may store location data 620 for any user interface element 614 generated in a keyboard location database. The keyboard location database 622 may store location data 620 for any application. The database 622 may also associate the location data 620 with metadata, such as a user identifier identifying the user whose action generated the location data 620, characteristics of the user (e.g., age, gender, height, etc.), an identifier or type of application (e.g., a web browser), the environment in which the application is being used (e.g., when the predictive input 616 is being displayed), the display size, position, and / or orientation of the application when the location data 620 was generated, and the like. In particular embodiments, future requests for access to user interface elements 614 for a particular application may query (e.g., based on information associated with the aforementioned metadata) keyboard position database 622 to generate a user interface element at a certain location based on position data 620 stored in keyboard position database 622. For example, if a particular user 101 is interacting with a particular application in a particular environment (e.g., when the application's panel size is reduced and positioned toward the bottom of the user's 101 field of view), such information may be used to query database 622 to find relevant position data 620, which may be used to automatically position the virtual keyboard, e.g., to minimize the likelihood that it obscures any content of interest.

[0058] The position data 620 may indicate a location where the user interface element 614 does not obstruct any content of the application of the panel 602. Since the content of the application may be helpful or critical in determining input to the user interface element 614, the client system 130 may identify a way to prevent obstruction of the application content. As an example and not by way of limitation, the reasons why the obstruction of content may prevent the user 101 from connecting with the application of the panel 602 may include that the content provides information to complete the field box (e.g., the question to be answered by the field box), the content may provide predictive input 616, and other reasons. Generating the user interface element 614 at a specific location may prevent obstruction of the content of the application. The position data 620 may indicate an area where the user 101 tends to move the user interface element 614 to interact with the application. In a particular embodiment, the position data 620 may include position data associated with other elements as described above, and is used to generate other elements at a location that does not obstruct the content of the user interface element 614. As an example and not by way of limitation, the position data 620 of the search field box 612 may be stored and used to determine a location to generate the search field box 612, thereby preventing obstruction of content, such as the predictive input 616 of the user interface element 614. In certain embodiments, the position data 620 of other elements can be used to generate and display the user interface element 614 at a location that avoids obstructing the content of the application. In certain embodiments, the position data 620 can be compiled for multiple users 101 of a particular application (e.g., a user community of a virtual reality or social networking platform) to compile the position data 620 of the keyboard position database 622. The compilation of the position data 620 can indicate the trend of where the user 101 most wants to display the user interface element 614. The area avoided by the user interface element 614 provided by the position data 620 can indicate the area containing content and the area to be avoided. Although the present disclosure discusses moving the user interface element 614 generally within a plane parallel to the panel 602, the user 101 can move the user interface element 614 anywhere within the virtual reality environment 600. As an example and not by way of limitation, the user 101 can move the user interface element in three degrees of freedom within the virtual reality environment 600.

[0059] FIG. 7A to FIG. 7B Another example process for maintaining a user interface position database while user 101 is participating in virtual reality environment 700 is shown. Fig. 7A User 101 is shown connecting to the application of panel 702. Similar to Figure 6B , the user 101 may have chosen to interact with the interactive field, and a dedicated plane 704 including a search field box 706 and a user interface element 708 may be generated. The search box 706 may include a plurality of predicted inputs 710. The client system 130 may also store the position data 712 in a keyboard position database 714. Fig. 7AAs shown, user 101 may use virtual reality input device 134 to perform gesture 716 indicating a direction to move user interface element 708. In certain embodiments, user 101 may need to input input (e.g., click a button) to virtual reality input device 134 to initiate the process of performing the gesture. In certain embodiments, user 101 may need to hover a pointer (not shown) over user interface element 708 and perform gesture 716.

[0060] Figure 7B The result of performing gesture 716 is shown, which may cause user interface element 708 to move a predetermined amount in the direction specified by gesture 716. Figure 6C , the client system 130 may store the position data 712 in a keyboard position database associated with the user interface element 708 and use the position data 712 in a similar manner. In certain embodiments, the client system 130 may store the position data 712 of other elements of the application. The position data 712 may be used to determine the position of the user interface element 708 that does not obstruct the content of the application 702. In certain embodiments, the user 101 may use a gesture of the virtual reality input device 134 to "push" the user interface element 708 (e.g., into the screen). In certain embodiments, the user interface element 708 may be attached to one virtual reality input device 134 in the virtual reality environment 700, and another virtual reality input device 134 may be used to interact with the user interface element 708. In certain embodiments, the virtual reality input device 134 may include a touchpad that may activate a radial menu for one or more quick options. The touchpad may be a virtual touchpad to be selected in the virtual reality environment 700. Alternatively, the touchpad may be a physical component of the virtual reality input device 134. The one or more quick options may include a plurality of options to be activated to perform various functions, such as returning to the home virtual reality environment, closing the virtual reality environment, and the like.

[0061] Figure 8An example method 800 for generating a user interface element in a virtual reality environment is shown. The method may begin at step 810, where a client system (e.g., a virtual reality system) may receive a request to access a user interface element (e.g., a keyboard generated by an application) in a virtual reality environment of an application. At step 820, the client system may generate a user interface element at a first location in the virtual reality environment, obscuring a portion of the application. For example, a keyboard may be generated for a user to connect to, which may cover the content of the application (e.g., text associated with a question that the user answers with the keyboard). At step 830, the client system may detect an input that moves the user interface element from a first location in the virtual reality environment to a second location in the virtual reality environment relative to a display area of ​​the application. For example, a user may click and hold on the keyboard to drag the keyboard to another location in the virtual reality environment. In another example, a user may perform a swipe gesture to move the keyboard from one location to another location. At step 840, the client system may store location data associated with the second location of the virtual reality environment in a location database dedicated to the application. For example, the client system may store keyboard location data for a particular application (e.g., Facebook). At step 850, the client system may receive another request to access a user interface element in the virtual reality environment of an application. At step 860, the client system may query the location database for application-specific location data. At step 870, the client system may generate a user interface element at a location in the virtual reality environment based on the application-specific location data. For example, the client system may determine a location that does not obstruct the content of the application in the virtual reality environment and display the user interface element at that location. Particular embodiments may repeat the steps described above where appropriate. Figure 8 Although the present disclosure will Figure 8 Certain steps of the method are described and shown as occurring in a particular order, but the present disclosure contemplates occurring in any suitable order. Figure 8 In addition, although the present disclosure describes and illustrates example methods (including Figure 8 ), but the present disclosure contemplates any suitable method (including any suitable steps) for generating user interface elements in a virtual reality environment, which steps may include, where appropriate: Figure 8 In addition, although the present disclosure describes and illustrates the implementation of Figure 8 specific components, devices or systems for specific steps of the method, but the present disclosure contemplates performing Figure 8 Any suitable combination of any suitable components, devices or systems of any suitable steps of the method.

[0062] 9A to 9H An example process of copying and pasting content in virtual reality environment 900 is shown. Fig. 9A The user 101 is shown wearing a virtual reality headset 132 and using a virtual reality input device 134 to interact with the virtual reality environment 600. Figures 2A to 2G , client system 130 may use virtual reality headset 132 to render a virtual reality environment including the above-described virtual reality elements. In certain embodiments, virtual reality environment 900 may include a panel 902 that displays an application selected by user 101. The application (e.g., Facebook) displayed in panel 902 may include a plurality of posts 904. Posts 904 may contain text, images, links to websites, and other content typically found on posts on online social networks. Fig. 9A It is shown that the user 101 can point the virtual reality input device 134 to the application post 904 with the pointer 906 and the pointer path 908. Fig. 9A As shown, user 101 may enclose content of post 904 along path 910 from first pointer location 906a to second pointer location 906b. In certain embodiments, client system 130 may continuously store position data of virtual reality input device 134 and determine when a gesture is made within a predetermined amount of time. As an example and not by way of limitation, client system 130 may monitor the position of pointer 906 on panel 902 and determine when a gesture is made to enclose content (e.g., loop, circle, etc.) within a short time frame (e.g., 3 seconds). In certain embodiments, user 101 may need to input input (e.g., click a button) to virtual reality input device 134 to initiate a gesture.

[0063] Fig. 9BThe result of performing a gesture and following the path 910 is shown, which may cause the content of the post 904 surrounded by the path 910 to be highlighted in the text box 912, and the option box 914 to appear. The option box 914 may include a plurality of options 916, such as "copy" 916a, "paste" 916b, and "resize copied text" 916c. In certain embodiments, the option box 914 may include other options 916 and remove some options 916. Optionally, in certain embodiments, a radial menu may be generated containing the options 916 on the virtual reality input device 134 located within the virtual reality environment 900. The content shown may include the text surrounded by the path 910 in the text box 912. Heuristics may be used to determine what content the gesture encompasses. In certain embodiments, if a word is partially surrounded, the word may be included in the text box 912. Although the present disclosure describes the content as text, the present disclosure contemplates pictures and other content of the application. As an example and not by way of limitation, the path 910 may at least partially surround an image, and the image may be selected to be copied to a temporary storage of the content.

[0064] Fig. 9C It is shown that the user 101 can select the "adjust the size of the copied text" option 916c by pointing the pointer 906 to the desired location and inputting an input into the virtual reality input device 134 (e.g., clicking a button). In certain embodiments, the selection of the "adjust the size of the copied text" option 916c can execute instructions to start the resizing process of the text box 912. Fig.9D The result of selecting option 916c is shown, which may allow user 101 to enter one or more inputs to determine the left and right sides of text box 912. Fig.9D As shown, user 101 may select a corner of text box 912 with pointer 918 via pointer path 920 at pointer location 918a, and move pointer 918 to a second pointer location 918b by following path 922. Fig.9D A specific method of resizing text box 912 is shown, but in certain embodiments, user 101 may simply select locations on either side of text box 912. In certain embodiments, user 101 may be required to maintain input to virtual reality input device 134 (e.g., hold down a button) while resizing text box 912. Fig.9EThe result of moving the pointer 918 from pointer position 918a to pointer position 918b is shown, which can resize the text box 912. In certain embodiments, the virtual reality input device 134 can be set to a resize mode, and movement of the analog stick or input in the directional pad can cause the text box 912 to resize one character at a time. The longer the user 101 holds the analog stick or a particular direction of the directional pad, the faster the text box 912 can resize (e.g., two characters per second).

[0065] Fig.9F The result of resizing text box 912 is shown, which may redisplay option box 914. In certain embodiments, the resizing process may end when user 101 releases input on virtual reality input device 134 (e.g., releases a button). In certain embodiments, the resizing process may end when user 101 inputs input to virtual reality input device 134 (e.g., clicks a button) to end the resizing process. Fig.9F As shown, user 101 may select option 916a to execute the copy command. The copy command may store the contents of text box 912 in a temporary memory. In certain embodiments, the copy command may also be executed on the image.

[0066] Figure 9G The user 101 is shown connected to a message thread 924 within the application of the panel 902. The message thread 924 may include the content of a previous conversation between the user 101 and another user 101 of the online social network. The message thread 924 may include a response box 926 for the user 101 to enter text to be entered into the message thread 924. Figure 9G As shown, option box 914 displays option 916. Option box 914 may be displayed in response to receiving input from virtual reality input device 134 to display option box 914. Figure 9G As shown, the user 101 may select the “Paste” option 916 b which will execute the paste instruction by pointing the pointer 906 to the desired location and inputting an input into the virtual reality input device 134 (e.g., clicking a button). Figure 9H The result of selecting option 916b is shown, which executes a paste function and enters the copied contents 926 of text box 912 retrieved from temporary storage.

[0067] Fig.10An example method 1000 for copying and pasting content in a virtual reality environment is shown. The method may begin at step 1010, where a client system (e.g., a virtual reality system) may receive location data associated with a gesture made by a user. As an example and not by way of limitation, the location data may be associated with a location of a pointer of a virtual reality input device projected onto a surface in the virtual reality environment. At step 1020, the client system may determine a path drawn on a surface in the virtual reality environment based on the projection of the location data in the virtual reality environment. For example, the client system may determine the location of the pointer of the virtual reality input device over the past few seconds. At step 1030, the client system may identify one or more words surrounded by the path, the one or more words displayed on a surface in the virtual reality environment. For example, the client system may identify the words surrounded by the path based on the location data of the path relative to the location data of the words. At step 1040, the client system may receive an instruction from the user to copy the one or more words surrounded by the path. At step 1050, the client system may receive input from the user indicating a location in the virtual reality environment. For example, the user may point to a comment box of a post on an online social network. At step 1060, the client system may receive another instruction from the user to paste the one or more words. At step 1070, the client system may display the one or more words that have been copied at the location indicated by the user. Where appropriate, certain embodiments may repeat Fig.10 Although the present disclosure will Fig.10 Certain steps of the method are described and shown as occurring in a particular order, but the present disclosure contemplates occurring in any suitable order. Fig.10 In addition, although the present disclosure describes and illustrates example methods (including Fig.10 ), but the present disclosure contemplates any suitable method (including any suitable steps) for copying and pasting content in a virtual reality environment, which steps may include, where appropriate: Fig.10 In addition, although the present disclosure describes and illustrates the implementation of Fig.10 specific components, devices or systems for specific steps of the method, but the present disclosure contemplates performing Fig.10 Any suitable combination of any suitable components, devices or systems of any suitable steps of the method.

[0068] FIG. 11A to FIG. 11C An example process for utilizing redirection mode in virtual reality environment 1100 is shown. Fig.11A User 101 is shown wearing virtual reality headset 132 in a bedroom environment. Figures 2A to 2G , the client system 130 can use the virtual reality headset 132 to render a virtual reality environment including the above-mentioned virtual reality elements. In certain embodiments, the user 101 can sit on the bed 1102 while seeing the virtual reality environment 1100, which can include generated virtual reality elements 1104 (e.g., a theater stage) and a panel 1106 that displays content and / or applications (e.g., a video) that the user 101 has selected. The panel 1106 can be fixed relative to a position in the virtual reality environment 1100. In certain embodiments, the panel 1106 can be coupled to an anchor point 1108, which is fixed relative to a position in the virtual reality environment 1100. As shown in FIG. Fig.11A As shown, user 101 may have: a center line 1110 of his field of view, which may be located in the middle of panel 1106; a top line 1112 of his field of view, creating an upper limit on the outermost content he can see; and a bottom line 1114 of his field of view, creating a lower limit on the outermost content he can see. As an example and not by way of limitation, user 101's field of view may be limited within virtual reality environment 1100, and user 101 may only see panel 1106 and partially see generated virtual reality elements 1104. In certain embodiments, anchor point 1108 may be transparent and is only shown for reference to panel 1106.

[0069] Although the panel 1106 is fixed in space (i.e., fixed relative to the virtual environment 1100), when the user turns his head, the virtual reality device can adjust the content displayed to the user accordingly as expected. For example, when the user's field of view moves to the left, the virtual reality device will render the corresponding part of the virtual environment 1100 to the left of the panel 1106 (from the user's reference point), potentially causing the right side of the panel 1106 to be outside the user's field of view. Therefore, unless the user looks forward at the panel 1106, the user may not see the entire panel 1106. In contrast, if the virtual environment 1100 is fixed relative to the user's head-mounted device, the user can see the entire panel 1106 regardless of his viewing direction. However, doing so will also cause other virtual reality elements (such as the theater stage 1104, the rows of seats in the virtual theater, other virtual moviegoers, etc.) to also track the user's head-mounted device and lose the sense of physical reality in the virtual environment. For example, no matter how the user turns his head, what is displayed will remain the same (e.g., the user will continue to see the panel 1106 directly in front of him).

[0070] In certain embodiments, user 101 may enable a redirection mode of panel 1106 to accommodate the user's physical viewing orientation (e.g., reclining) without affecting the rest of the user's virtual reality experience. As an example and not by way of limitation, user 101 may enable redirection mode by pressing a button on virtual reality input device 134 (not shown). As another example and not by way of limitation, user 101 may press a virtual reality button to enable redirection mode by moving a pointer of virtual reality input device 134 toward the button and inputting input into virtual reality input device 134 (e.g., clicking the button). For example, anchor point 1108 may be a button that enables redirection mode.

[0071] Fig. 11B The result of enabling the redirection mode of the panel 1106 is shown, which can allow the user 101 to change position, and the panel 1106 is along the centerline 1110 of the user's field of view. In certain embodiments, the generated virtual reality element 1104 can stay in the same position within the virtual reality environment 1100, but the panel 1106 can follow the user's head movement. The client system 130 can receive sensor data indicating the change in the user's 101 viewpoint and redirect the panel 1106 based on the sensor data. The sensor data can be received from the virtual reality headset 132. In certain embodiments, the sensor data can include sensor data generated by any number of accelerometers, gyroscopes, magnetometers, and eye tracking sensors. As shown in FIG. Fig. 11B As shown, user 101 may continue path 1116 to lie down on his bed 1102 .

[0072] Fig. 11C The result of user 101 continuing to move along path 1116 is shown, which may cause panel 1106 to redirect itself in front of user 101 above bed 1102. In particular embodiments, user 101 can disable redirection mode, similar to how user 101 enabled redirection mode. Disabling of redirection mode can fix panel 1106 relative to this new position in virtual reality environment 1100. In particular embodiments, disabling of redirection mode can be the result of releasing a button on virtual reality input device 134 (e.g., where redirection mode requires the user to press and hold a button). In particular embodiments, anchor point 1108 is fixed relative to the new position in virtual reality environment 1100. Fig. 11CAs shown, user 101 may not be able to see the generated virtual reality element 1104 (e.g., a theater stage) due to his own redirection into the bed, which is a realistic, expected result when looking up. In a similar manner, the user can use the redirection mode to adjust the content display panel to be attached to the virtual reality environment in any desired position / orientation to accommodate any desired physical viewing direction of the user (e.g., lying down, reclining, etc.), while maintaining the realism of the virtual reality experience once the panel is set up. Although only one panel 1106 is shown, the present disclosure contemplates having multiple panels coupled to anchor points that can be redirected similarly to panel 1106. In certain embodiments, by using multiple panels, user 101 can selectively redirect specific panels within the virtual reality environment 1100.

[0073] FIG. 12A to FIG. 12E An example process for utilizing a redirection mode in a virtual reality environment 1200 generated inside a moving vehicle 1202 is shown. Fig. 12A 1 shows a user 101 in a vehicle 1202 on the passenger side 1204 in a virtual reality environment 1200. That is, the user 101 is using a virtual reality headset 132 in the vehicle 1202. Figures 2A to 2G , client system 130 may use virtual reality headset 132 to render a virtual reality environment including the above-mentioned virtual reality elements. Vehicle 1202 may be driven along path 1206 and prepare to turn. In certain embodiments, virtual reality environment 1200 may include generated virtual reality elements 1208 (e.g., a theater stage) and panel 1210 content and / or applications (e.g., a video) that user 101 has selected. Panel 1210 may be fixed relative to a position in virtual reality environment 1200. Initial position 1212 may be shown as a reference to the initial orientation of user 101 in virtual reality environment 1200. As shown in FIG. Fig. 12A As shown, user 101 can have: a center line 1214 of his field of view, which can be located in the middle of panel 1210; a right line 1216 of his field of view, creating a right boundary about the outermost content he can see; and a left line 1218 of his field of view, creating a left boundary about the outermost content he can see. These three different lines 1214, 1216, 1218 can create a field of view 1220 of user 101.

[0074] One problem with using a virtual reality device in a moving vehicle is that the movement of the vehicle (e.g., acceleration, turning, vibration, etc.) may affect the content that a virtual reality application determines should be displayed to the user. For example, when a vehicle turns a corner, the sensors of the virtual reality device (e.g., inertial measurement unit, gyroscope, accelerometer, etc.) may detect that a turn has been made and, in response, move the user's virtual viewpoint, even though the user has not moved relative to the vehicle. Thus, even though the user may expect to see the same scene, the scene may drift due to the movement of the vehicle. This effect may degrade the user's experience or even render an application effectively unusable in a moving vehicle.

[0075] In certain embodiments, user 101 may enable a redirection mode for traveling with virtual reality headset 132. A "travel mode" may dynamically adjust and redirect virtual reality environment 1200 in response to detected movement to realign generated virtual reality elements 1208 and panels 1210 back to an initial orientation indicated by initial position 1212. In certain embodiments, enabling of the redirection mode for traveling may capture the initial orientation of generated virtual reality elements 1208 and panels 1210. User 101 may enable a redirection mode for traveling, similar to enabling of a redirection mode for virtual reality panel 1106 as described above.

[0076] FIG. 12A to FIG. 12E An example process for utilizing the redirection mode in a virtual reality environment 1200 generated inside a mobile vehicle 1202 is shown. Fig. 12A As shown, as vehicle 1202 turns along path 1206 , generated virtual reality elements 1208 and panel 1210 may follow path 1222 .

[0077] Fig. 12B The result of vehicle 1202 following path 1206 is shown. The movement of the vehicle may cause portions of the generated virtual reality elements 1208 to be redirected outside of the field of view of user 1220 (because the virtual reality device may think the user's head is turning). This may be the result of client system 130 experiencing inertial motion or acceleration due to vehicle 1202 performing turns along path 1206. Client system 130 may be receiving sensor data that continuously indicates changes in direction, and client system 130 may erroneously determine the sensor data as a change in position of user 101 and redirect virtual reality environment 1200 as a result. Initial position 1212 may remain in the same location to show that virtual reality environment 1200 is redirected due to the sensor data. As shown in FIG. Fig. 12B As shown, vehicle 1202 may continue along path 1206 to complete the turn, and similarly, generated virtual reality element 1208 may continue to be redirected outside of user field of view 1220.

[0078] Fig. 12C The result of vehicle 1202 completing a turn along path 1206 is shown, which further moves generated virtual reality elements 1208 and 1210 to the boundary of user 101's view left line 1218. Generated virtual reality element 1208 may be further pushed outside of user 101's field of view 1220. Fig. 12C As shown, vehicle 1202 may continue along path 1206, which is not straight, and generated virtual reality element 1208 and panel 1210 may redirect themselves along initial position 1212 to return to the initial orientation.

[0079] Fig.12D The result of the vehicle 1202 following the path 1206 is shown, wherein the generated virtual reality element 1208 and the panel 1210 begin to realign with the initial position 1212 to return to the initial orientation. As a result of the realignment with the initial position 1212, the generated virtual reality element 1208 may begin to return to the field of view 1220 of the user 101. Fig.12D As shown, vehicle 1202 may continue along path 1206 , and generated virtual reality elements 1208 and panel 1210 may continue to realign with initial position 1212 .

[0080] Fig.12EThe result of the vehicle 1202 following the path 1206 is shown, wherein the generated virtual reality elements 1208 and panels 1210 are realigned with the initial position 1212 and returned to the initial orientation. In certain embodiments, the client system 130 may determine that the change in orientation is due to the motion of the vehicle 1202, and in response to determining that the change in orientation is due to the motion of the vehicle 1202, perform a readjustment. As an example and not by way of limitation, the client system 130 may receive sensor data indicating a change in orientation, and may further determine that the change in orientation may be attributed to vehicle motion of the vehicle 1202 that the user 101 is occupying. To determine that the change in orientation is due to vehicle motion rather than user motion, the client system 130 may receive information such as the geographic location of the virtual reality headset 132 and the corresponding user 101. As an example and not by way of limitation, if the client system 130 determines that the geographic location has changed significantly within a predetermined amount of time, it may be assumed that the user 101 is in a moving vehicle 1202, and a readjustment of orientation may be performed as described above. In certain embodiments, the client system 130 may receive information indicating a turning speed of the client system 130. As another example and not by way of limitation, if the client system 130 determines that the turning speed is below a threshold turning speed, it may be assumed that the user 101 is in a vehicle (e.g., an airplane) and a re-adjustment of direction may be performed as described above. In certain embodiments, the client system 130 may receive information indicating a global positioning system (GPS) signal of the client system 130 and information indicating a travel route of the vehicle 1202 (e.g., navigation data). As an example and not by way of limitation, the client system 130 may determine when bends and turns may occur within the travel route and dynamically re-adjust the direction of the virtual reality environment 1200 by taking into account the effect that the bends and / or turns will affect the direction of the virtual reality environment 1200 as described above. For example, by using the speed of the vehicle and the direction in which the vehicle is traveling, adjustments due to vehicle motion may be performed in conjunction with the time at which they may occur on the travel route. In certain embodiments, motion sensor data (e.g., detected by an inertial measurement unit) may be compared to GPS data to assess the likelihood that the motion sensor data is attributable to vehicle motion. For example, if the motion sensor data indicates that a turn is being made and the GPS data also indicates a turn, the system may infer that the detected motion is due to the moving vehicle and not the user. On the other hand, if the GPS data indicates that the user is stationary or moving in a manner inconsistent with the motion sensor data (for example, the GPS data may indicate that the user is moving in a straight line or turning in a direction that is different from the turn detected by the inertial measurement unit), the detected motion can be attributed to user movement and the device can respond accordingly.In certain embodiments, readjusting the user's 101 field of view 1220 back to the initial direction may be accomplished within a predetermined time interval (e.g., 3 seconds) so that the virtual reality environment 1200 does not change too quickly and cause motion sickness to the user 101 and / or change too slowly and cause inconvenience to the user 101.

[0081] Fig.13 An example method 1300 for utilizing a redirection mode in a virtual reality environment is shown. The method may begin at step 1310, where a client system (e.g., a virtual reality system) may generate a virtual reality panel to display content in a virtual reality environment. The virtual reality panel may be fixed relative to a location in the virtual reality environment. As an example and not by way of limitation, the client system may generate a video display or a web page display at the center of a virtual reality theater. At step 1320, the client system may receive input to enable a first redirection mode of the virtual reality panel in the virtual reality environment. The first redirection mode may allow the virtual reality panel to be redirected relative to a user's viewpoint. At step 1330, the client system may receive sensor data indicating a change in the user's viewpoint. For example, sensor data of an accelerometer, a gyroscope, a magnetometer, and an eye tracking sensor may be collected and processed by the client system to determine whether the user's viewpoint has changed. At step 1340, the client system may redirect the virtual reality panel based on the received sensor data. For example, the virtual reality panel may follow the user's viewpoint. At step 1350, when the virtual reality panel is at a new location in the virtual reality environment, the client system may receive input to disable the first redirection mode. Disabling the first redirection mode may fix the virtual reality panel relative to the new location in the virtual reality environment. Where appropriate, specific examples may be repeated Fig.13 Although the present disclosure will Fig.13 Certain steps of the method are described and shown as occurring in a particular order, but the present disclosure contemplates occurring in any suitable order. Fig.13 Furthermore, although the present disclosure describes and illustrates example methods (including Fig.13 ), but the present disclosure contemplates any suitable method (including any suitable steps) for utilizing the redirection mode in a virtual reality environment, which may include, where appropriate, Fig.13 In addition, although the present disclosure describes and illustrates the implementation of Fig.13 specific components, devices or systems for specific steps of the method, but the present disclosure contemplates performing Fig.13 Any suitable combination of any suitable components, devices or systems of any suitable steps of the method.

[0082] Fig.14 An example computer system is shown. In a particular embodiment, one or more computer systems 1400 perform one or more steps of one or more methods described or shown herein. In a particular embodiment, one or more computer systems 1400 provide functionality described or shown herein. In a particular embodiment, software running on one or more computer systems 1400 performs one or more steps of one or more methods described or shown herein, or provides functionality described or shown herein. A particular embodiment includes one or more portions of one or more computer systems 1400. Herein, references to a computer system may include a computing device, and vice versa, where appropriate. Moreover, references to a computer system may include one or more computer systems, where appropriate.

[0083] The present disclosure contemplates any suitable number of computer systems 1400. The present disclosure contemplates computer systems 1400 in any suitable physical form. By way of example, and not by way of limitation, computer system 1400 may be an embedded computer system, a system on a chip (SOC), a single board computer system (SBC) (such as, for example, a computer module (COM) or a system module (SOM)), a desktop computer system, a laptop or notebook computer system, an interactive kiosk, a mainframe, a grid of computer systems, a mobile phone, a personal digital assistant (PDA), a server, a tablet computer system, or a combination of two or more thereof. Where appropriate, computer system 1400 may include one or more computer systems 1400; may be unitary or distributed; across multiple locations; across multiple machines; across multiple data centers; or located in a cloud, which may include one or more cloud components in one or more networks. Where appropriate, one or more computer systems 1400 may perform one or more steps of one or more methods described or illustrated herein without substantial spatial or temporal limitations. By way of example, and not by way of limitation, one or more computer systems 1400 may perform one or more steps of one or more methods described or illustrated herein in real time or in batches. Where appropriate, one or more computer systems 1400 may perform at different times or at different locations one or more steps of one or more methods described or illustrated herein.

[0084] In a particular embodiment, computer system 1400 includes a processor 1402, memory 1404, storage 1406, an input / output (I / O) interface 1408, a communication interface 1410, and a bus 1412. Although this disclosure describes and shows a particular computer system having particular numbers of particular components in a particular configuration, this disclosure contemplates any suitable computer system having any suitable numbers of any suitable components in any suitable configuration.

[0085] In a particular embodiment, the processor 1402 includes hardware for executing instructions, such as those that constitute a computer program. As an example, not by way of limitation, to execute instructions, the processor 1402 may retrieve (or fetch) instructions from an internal register, an internal cache, a memory 1404, or a storage 1406; decode these instructions and execute these instructions; and then write one or more results to an internal register, an internal cache, a memory 1404, or a storage 1406. In a particular embodiment, the processor 1402 may include one or more internal caches for data, instructions, or addresses. The present disclosure contemplates that the processor 1402 includes any suitable number of any suitable internal caches where appropriate. As an example, not by way of limitation, the processor 1402 may include one or more instruction caches, one or more data caches, and one or more translation lookaside buffers (TLBs). The instructions within the instruction cache may be copies of the instructions within the memory 1404 or the storage 1406, and the instruction cache may speed up the retrieval of those instructions by the processor 1402. The data in the data cache may be a copy of data in the memory 1404 or storage 1406 for the instructions executed on the processor 1402 to be acted upon; the results of previous instructions executed on the processor 1402 are accessed or written into the memory 1404 or storage 1406 by subsequent instructions executed on the processor 1402; or other suitable data. The data cache may speed up the read or write operations of the processor 1402. The TLB may speed up the virtual address translation of the processor 1402. In a particular embodiment, the processor 1402 may include one or more internal registers for data, instructions, or addresses. Where appropriate, the present disclosure contemplates that the processor 1402 includes any suitable number of any suitable internal registers. Where appropriate, the processor 1402 may include one or more arithmetic logic units (ALUs); be a multi-core processor; or include one or more processors 1402. Although the present disclosure describes and shows a particular processor, the present disclosure contemplates any suitable processor.

[0086] In a particular embodiment, the memory 1404 includes a main memory for storing instructions of the processor 1402 to be executed or data of the processor 1402 to be acted upon. As an example, and not by way of limitation, the computer system 1400 may load instructions from the storage 1406 or another source (e.g., another computer system 1400) into the memory 1404. The processor 1402 may then load the instructions from the memory 1404 into an internal register or an internal cache. To execute these instructions, the processor 1402 may retrieve the instructions from the internal register or the internal cache and decode these instructions. During or after executing the instructions, the processor 1402 may write one or more results (which may be intermediate or final results) into the internal register or the internal cache. The processor 1402 may then write one or more of these results into the memory 1404. In a particular embodiment, the memory 1404 executes only instructions in one or more internal registers or internal caches or in the memory 1404 (as opposed to storage 1406 or elsewhere), and operates only on data in one or more internal registers or internal caches or in the memory 1404 (as opposed to storage 1406 or elsewhere). One or more memory buses (which may include an address bus and a data bus) may couple the processor 1402 to the memory 1404. As described below, the bus 1412 may include one or more memory buses. In a particular embodiment, one or more memory management units (MMUs) are located between the processor 1402 and the memory 1404 and facilitate access to the memory 1404 requested by the processor 1402. In a particular embodiment, the memory 1404 includes a random access memory (RAM). Where appropriate, the RAM may be a volatile memory. Where appropriate, the RAM may be a dynamic RAM (DRAM) or a static RAM (SRAM). Moreover, where appropriate, the RAM may be a single-port or multi-port RAM. The present disclosure contemplates any suitable RAM. Where appropriate, memory 1404 may include one or more memories 1404. Although this disclosure describes and illustrates particular memories, this disclosure contemplates any suitable memories.

[0087] In a particular embodiment, storage 1406 includes a large capacity storage for data or instructions. As an example, and not by way of limitation, storage 1406 may include a HDD, a floppy disk drive, a flash memory, an optical disk, a magneto-optical disk, a tape, or a universal serial bus (USB) drive, or a combination of two or more thereof. Where appropriate, storage 1406 may include a removable or fixed (or fixed) medium. Where appropriate, storage 1406 may be located inside or outside of computer system 1400. In a particular embodiment, storage 1406 is a non-volatile solid-state memory. In a particular embodiment, storage 1406 includes a read-only memory (ROM). Where appropriate, the ROM may be a mask-programmed ROM, a programmable ROM (PROM), an erasable PROM (EPROM), an electrically erasable PROM (EEPROM), an electrically rewritable ROM (EAROM), or a flash memory, or a combination of two or more thereof. The present disclosure contemplates the use of a large capacity storage 1406 in any suitable physical form. Where appropriate, storage 1406 may include one or more storage control units that facilitate communication between processor 1402 and storage 1406. Where appropriate, storage 1406 may include one or more storages 1406. Although this disclosure describes and illustrates particular storage, this disclosure contemplates any suitable storage.

[0088] In a particular embodiment, the I / O interface 1408 includes hardware, software, or both, which provides one or more interfaces for communicating between the computer system 1400 and one or more I / O devices. Where appropriate, the computer system 1400 may include one or more of these I / O devices. One or more of these I / O devices can communicate between an individual and the computer system 1400. As an example, and not by way of limitation, the I / O device may include a keyboard, a key, a microphone, a display, a mouse, a printer, a scanner, a speaker, a still camera, a stylus, a tablet, a touch screen, a trackball, a video camera, another suitable I / O device, or a combination of two or more thereof. The I / O device may include one or more sensors. The present disclosure contemplates any suitable I / O device and any suitable I / O interface 1408 for these I / O devices. Where appropriate, the I / O interface 1408 may include one or more device or software drivers that can allow the processor 1402 to drive one or more of these I / O devices. Where appropriate, the I / O interface 1408 may include one or more I / O interfaces 1408. Although this disclosure describes and illustrates a particular I / O interface, this disclosure contemplates any suitable I / O interface.

[0089] In particular embodiments, the communication interface 1410 includes hardware, software, or both that provides one or more interfaces for communicating (such as, for example, packet-based communications) between the computer system 1400 and one or more other computer systems 1400 or one or more networks. By way of example, and not by way of limitation, the communication interface 1410 may include a network interface controller (NIC), or a network adapter for communicating with an Ethernet or other wired network or a wireless NIC (WNIC), or a network adapter for communicating with a wireless network (such as a WI-FI network). The present disclosure contemplates any suitable network and any suitable communication interface 1410 for the network. By way of example, and not by way of limitation, the computer system 1400 may communicate with one or more portions of an ad hoc network, a personal area network (PAN), a local area network (LAN), a wide area network (WAN), a metropolitan area network (MAN), or the Internet, or a combination of two or more thereof. One or more portions of one or more of these networks may be wired or wireless. As an example, the computer system 1400 may communicate with a wireless PAN (WPAN) (such as, for example, a BLUETOOTH WPAN), a WI-FI network, a WI-MAX network, a cellular telephone network (such as, for example, a Global System for Mobile Communications (GSM) network), or other suitable wireless networks, or a combination of two or more thereof. Where appropriate, the computer system 1400 may include any suitable communication interface 1410 for any of these networks. Where appropriate, the communication interface 1410 may include one or more communication interfaces 1410. Although the present disclosure describes and shows particular communication interfaces, the present disclosure contemplates any suitable communication interface.

[0090] In a particular embodiment, bus 1412 includes hardware, software, or both that couples components of computer system 1400 to each other. By way of example, and not by way of limitation, bus 1412 may include an accelerated graphics port (AGP) or other graphics bus, an enhanced industry standard architecture (EISA) bus, a front side bus (FSB), a hypertransport (HT) interconnect, an industry standard architecture (ISA) bus, an infinite bandwidth interconnect, a low pin position (LPC) bus, a memory bus, a micro channel architecture (MCA) bus, a peripheral component interconnect (PCI) bus, a PCI Express (PCIe) bus, a serial advanced technology attachment (SATA) bus, a video electronics standard association local (VLB) bus, or another suitable bus or a combination of two or more thereof. Where appropriate, bus 1412 may include one or more buses 1412. Although the present disclosure describes and shows a particular bus, the present disclosure contemplates any suitable bus or interconnect.

[0091] Herein, where appropriate, a computer-readable non-transitory storage medium or media may include a semiconductor-based or other integrated circuit (IC) (such as, for example, a field programmable gate array (FPGA) or an application-specific IC (ASIC)), a hard disk drive (HDD), a hybrid hard disk drive (HHD), an optical disk, an optical disk drive (ODD), a magneto-optical disk, a magneto-optical drive, a floppy disk, a floppy disk drive (FDD), a magnetic tape, a solid-state drive (SSD), a RAM drive, a secure digital card, a secure digital card or drive, another suitable computer-readable non-transitory storage medium, or any suitable combination of two or more thereof. Where appropriate, a computer-readable non-transitory storage medium may be volatile, non-volatile, or a combination of volatile and non-volatile.

[0092] As used herein, unless expressly specified otherwise or the context clearly dictates otherwise, "or" has an inclusive rather than exclusive meaning. Thus, as used herein, "A or B" means "A and / or B" unless expressly specified otherwise or the context clearly dictates otherwise. Furthermore, "and" has a joint and separate meaning unless expressly specified otherwise or the context clearly dictates otherwise. Thus, as used herein, "A and B" means "A and B jointly or separately" unless expressly specified otherwise or the context clearly dictates otherwise.

[0093] The scope of the present disclosure includes all changes, substitutions, variations, alterations and modifications of the example embodiments described or shown in this article as understood by those skilled in the art. The scope of the present disclosure is not limited to the example embodiments described or shown in this article. Moreover, although the present disclosure describes and shows that the corresponding embodiments in this article include specific components, elements, features, functions, operations or steps, any of these embodiments may include any combination or arrangement of any components, elements, features, functions, operations or steps described or shown anywhere in this article as understood by those skilled in the art. In addition, as long as the device, system, or device or system component that is suitable for, set to, can be, configured to, enabled, can be used for or effectively performs a specific function is suitable, set, can, configured, enabled, available or effective, regardless of whether the device, system, component or the specific function is activated, opened or turned on, the device, system, component or the specific function is cited in the attached claims, or the system, or the component includes the device, system, component. In addition, although the present disclosure describes or shows a specific embodiment as providing a specific advantage, a specific embodiment may not provide these advantages, provide some or all of these advantages.

Claims

1. A method for generating a user interface element in a virtual reality environment, comprising: receiving a first request for a first application to access a user interface element in a virtual reality environment; The user interface element is generated at a first location in the virtual reality environment, wherein: The user interface element at the first position obscures a portion of the first application; detecting input to move the user interface element from the first position in the virtual reality environment to a last position in the virtual reality environment relative to a display area of ​​the first application; storing position data in a position database based on the last position in the virtual reality environment, wherein the stored position data: (1) corresponds to a spatial relationship between the user interface element and the first application, and (2) is associated with the first application among a plurality of applications on the client system, wherein the storing of the position data includes storing the position data for each of the plurality of applications; receiving a second request to access the user interface element in the virtual reality environment; determining that the user interface element requested by the second request is associated with the first application; querying the location database to select previously stored location data associated with the first application; and The user interface element is generated at a location in the virtual reality environment based on previously stored position data associated with the first application.

2. The method according to claim 1, wherein: The stored position data associated with the first application includes multiple inputs to move the user interface element from the first position in the virtual reality environment to the last position in the virtual reality environment relative to the display area of ​​the first application for multiple users.

3. The method according to claim 1, wherein: The user interface elements include at least a keyboard.

4. The method according to claim 1, wherein: The portion of the first application obscured by the user interface element includes content of the first application.

5. The method according to claim 4, wherein: The content of the first application includes predicted input generated in response to receiving a request to access the user interface element.

6. The method according to claim 5, wherein: The predicted input includes a most recent input into the user interface element.

7. The method according to claim 4, wherein: Moving the user interface element includes moving the user interface element to a position that avoids obstructing content of the first application.

8. The method of claim 4, further comprising repositioning content of the first application relative to a user interface element at the first location in the virtual reality environment to avoid obstructing content of the first application.

9. The method according to claim 4, further comprising: retrieving content data indicating a location of content relative to the display area of ​​the first application; determining a position of the user interface element that avoids content of the first application; and The user interface element is generated at a location that avoids content of the first application.

10. The method according to claim 1, wherein: The input to move the user interface element comprises manual selection of the user interface element from the first position to the final position.

11. The method according to claim 1, wherein: The input to move the user interface element includes a swipe gesture in one direction, wherein the direction of the swipe gesture is from the first position to the last position, and wherein the user interface element moves a predetermined amount of space relative to the display area of ​​the first application.

12. The method according to claim 1, wherein: Moving the user interface element includes moving the user interface element in a plane parallel to the display area of ​​the first application.

13. The method according to claim 1, wherein: Moving the user interface element includes moving the user interface element to any place in the virtual reality environment.

14. The method according to claim 1, wherein: Storing the location data includes storing the location data for each of the plurality of applications.

15. The method according to claim 1, further comprising: receiving a request for a second application to access the user interface element in the virtual reality environment; generating the user interface element at a third position in the virtual reality environment, wherein the user interface element at the third position obscures a portion of the second application; detecting an input to move the user interface element from the third position in the virtual reality environment to a fourth position in the virtual reality environment relative to a display area of ​​the first application; storing position data associated with the fourth position of the virtual reality environment in the position database, wherein the stored position data associated with the fourth position is associated with the second application; receiving another request for the second application to access the user interface element in the virtual reality environment; querying the location database for stored location data associated with the second application; and The user interface element is generated at a location in the virtual reality environment based on stored position data associated with the second application.

16. The method according to claim 1, wherein: Querying the location database for stored location data associated with the first application includes querying the stored location data associated with the first application for a user.

17. One or more computer-readable non-transitory storage media containing software that, when executed, is operable to: receiving a first request for a first application to access a user interface element in a virtual reality environment; The user interface element is generated at a first location in the virtual reality environment, wherein: The user interface element at the first position obscures a portion of the first application; detecting input that moves the user interface element from the first position in the virtual reality environment to a last position of the user interface element in the virtual reality environment relative to a display area of ​​the first application; storing position data in a position database based on the last position of the user interface element in the virtual reality environment, wherein the stored position data: (1) corresponds to a spatial relationship between the user interface element and the first application, and (2) is associated with the first application among a plurality of applications on a client system, wherein the storing of the position data includes storing the position data for each of the plurality of applications; receiving a second request to access the user interface element in the virtual reality environment; determining that the user interface element requested by the second request is associated with the first application; querying the location database to select previously stored location data associated with the first application; and The user interface element is generated at a location in the virtual reality environment based on previously stored position data associated with the first application.

18. The medium according to claim 17, wherein: The stored position data associated with the first application includes multiple inputs to move the user interface element from the first position in the virtual reality environment to the last position in the virtual reality environment relative to the display area of ​​the first application from multiple users.

19. A system for generating user interface elements in a virtual reality environment, comprising: one or more processors; and a non-transitory memory coupled to the processor, the non-transitory memory comprising instructions executable by the processor, the processor, when executing the instructions, being operable to: receiving a first request for a first application to access a user interface element in a virtual reality environment; generating the user interface element at a first location in the virtual reality environment, wherein the user interface element at the first location obscures a portion of the first application; detecting input that moves the user interface element from the first position in the virtual reality environment to a last position of the user interface element in the virtual reality environment relative to a display area of ​​the first application; storing position data in a position database based on the last position of the user interface element in the virtual reality environment, wherein the stored position data: (1) corresponds to a spatial relationship between the user interface element and the first application, and (2) is associated with the first application among a plurality of applications on a client system, wherein the storing of the position data includes storing the position data for each of the plurality of applications; receiving a second request to access the user interface element in the virtual reality environment; determining that the user interface element requested by the second request is associated with the first application; querying the location database to select previously stored location data associated with the first application; and The user interface element is generated at a location in the virtual reality environment based on previously stored position data associated with the first application.

20. The system of claim 19, wherein: The stored position data associated with the first application includes multiple inputs to move the user interface element from the first position in the virtual reality environment to the last position in the virtual reality environment relative to the display area of ​​the first application from multiple users.

Images