Methods and apparatus for aligning components in a virtual reality environment

Abstract

The present invention relates to methods and apparatus for aligning components in a virtual reality environment. An example method includes: detecting a first input from a handheld controller of a virtual reality system; in response to detecting the first input, instructing a user to orient the handheld controller in a specified direction; detecting a second input from the handheld controller; and in response to detecting the second input, storing alignment data representing the alignment of the handheld controller.

Description

[0001] Case Analysis

[0002] This application is a divisional application of Chinese Invention Patent Application No. 201780011132.6, filed on May 15, 2017.

[0003] Cross-references to related applications

[0004] This application is a continuation of and claims priority to U.S. non-provisional patent application No. 15 / 594,186, filed May 12, 2017, entitled “METHODS AND APPARATUS TO ALIGN COMPONENTS IN VIRTUAL REALITY ENVIRONMENTS,” which claims priority to U.S. provisional patent application No. 62 / 335,829, filed May 13, 2016, entitled “METHODS AND APPARATUS TO ALIGN COMPONENTS IN VIRTUAL REALITY ENVIRONMENTS,” the disclosure of which is incorporated herein by reference in its entirety.

[0005] This application also claims priority to U.S. Provisional Patent Application No. 62 / 335,829, filed May 13, 2016, the disclosure of which is incorporated herein by reference in its entirety. Technical Field

[0006] This disclosure generally relates to virtual reality (VR) environments, and more specifically, to methods and apparatus for aligning components in a VR environment. Background Technology

[0007] Virtual reality (VR) systems generate immersive virtual environments for users. For example, an immersive environment can be three-dimensional (3D) and may include multiple virtual objects that the user can interact with. Users can experience immersive virtual environments via various display devices, such as monitors, glasses, goggles, helmets, or other head-mounted devices that the user views while viewing the display.

[0008] Once immersed in a 3D virtual environment, users can traverse and move to other areas of the virtual environment by physically moving and / or manipulating electronic devices to interact with and personalize their interactions. For example, a VR system may include sensors for tracking the user's head or body movements. Additionally, a VR system may include a handheld device that includes sensors and other such electronic components. Users can use this handheld device to navigate and interact with the virtual environment. Summary of the Invention

[0009] Some aspects of the method include: detecting a first input from a handheld controller of a virtual reality system; in response to detecting the first input, instructing a user to orient the handheld controller in a specified direction; detecting a second input from the handheld controller; and in response to detecting the second input, storing alignment data representing the alignment of the handheld controller.

[0010] Some aspects include a computer program product comprising a non-transitory storage medium, the computer program product including code that, when executed by processing circuitry of a virtual reality system configured to generate a virtual reality environment, causes the processing circuitry to perform a method comprising: detecting a first input from a handheld controller of the virtual reality system; in response to detecting the first input: communicatively coupling the handheld controller to a head-mounted display of the virtual reality system and instructing a user to orient the handheld controller in a specified direction; detecting a second input from the handheld controller; and in response to detecting the second input, storing alignment data representing the alignment of the handheld controller.

[0011] Some aspects include an electronic device configured to generate a virtual reality environment, the electronic device including a memory and control circuitry coupled to the memory, the control circuitry being configured to: detect a first input from a handheld controller of a virtual reality system; in response to detecting the first input: communicatively couple the handheld controller to a head-mounted display of the virtual reality system via a wireless signal and instruct a user to orient the handheld controller in a specified direction; detect a second input from the handheld controller; and in response to detecting the second input: determine the controller orientation of the handheld controller, determine the display orientation of the head-mounted display; and store alignment data representing the alignment of the handheld controller.

[0012] Some aspects of the method include: detecting input from a handheld controller of a virtual reality system; communicatively coupling the handheld controller to a head-mounted display of the virtual reality system in response to the input; displaying a first symbol associated with the handheld controller and a second symbol associated with the head-mounted display in a virtual reality scene in response to the communicative coupling of the handheld controller to the head-mounted display; and storing data indicating alignment of the handheld controller with the head-mounted display when the first symbol at least partially overlaps with the second symbol in the virtual reality scene.

[0013] Some aspects of the method include: detecting input from a handheld controller of a virtual reality system; communicatively coupling the handheld controller to a head-mounted display of the virtual reality system in response to the input; instructing a user to orient the head-mounted display in a specified direction while the input is detected; displaying an indicator representing the orientation of the head-mounted display in a virtual reality scene; and storing data representing the alignment of the head-mounted display when no more input is detected.

[0014] According to one aspect, a computer-readable medium has recorded and specifically implemented instructions thereon, which, when executed by a processor of a computer system, cause the computer system to perform any of the methods and processes disclosed herein.

[0015] Details of one or more embodiments are set forth in the following drawings and description. Other features will be apparent from this specification and the drawings, and from the claims. Attached Figure Description

[0016] Figure 1 This is a block diagram of an example VR system that can be used to align components in a VR environment, according to this disclosure.

[0017] Figure 2 This is a schematic diagram illustrating an example head-mounted display (HMD) according to the teachings of this disclosure.

[0018] Figure 3 This is a schematic diagram illustrating an example handheld controller according to the teachings of this disclosure.

[0019] Figure 4 This is a flowchart representing example methods that can be used to implement HMD.

[0020] Figure 5 This is a flowchart illustrating example methods that can be used to implement a handheld controller.

[0021] Figure 6 This is a sample VR scene showing an example target and an example cursor.

[0022] Figure 7 This is a sample interaction of a sample VR scene that shows a sample target and a sample cursor.

[0023] Figure 8A and Figure 8B The diagram sequentially illustrates example alignments of the HMD with the VR scene.

[0024] Figure 9 This is another flowchart representing example methods that can be used to implement HMD.

[0025] Figure 10 This is another flowchart representing example methods that can be used to implement HMD.

[0026] Figure 11 This is an example VR scene showing an HMD and a handheld controller.

[0027] Figure 12 These are block diagrams of example computer devices and example mobile computer devices that can be used to implement the examples disclosed herein. Detailed Implementation

[0028] Reference will now be made in detail to non-limiting examples of this disclosure, which are illustrated in the accompanying drawings. The examples are described below with reference to the accompanying drawings, wherein similar reference numerals denote similar elements. Where similar reference numerals are shown, corresponding descriptions are not repeated, and interested readers can refer to the previously discussed figures for a description of similar elements.

[0029] Turning Figure 1 This diagram illustrates a block diagram of an example VR system 100 for creating and interacting with a three-dimensional (3D) virtual reality (VR) environment in accordance with the teachings of this disclosure. Generally, system 100 provides a 3D VR environment and VR content that enables a user to access, view, use, and / or interact with the examples described herein. System 100 may provide the user with options for accessing content, applications, virtual objects, and VR controls using, for example, eye gaze and / or movement within the VR environment. Figure 1 The example VR system 100 includes a head-mounted display (HMD) 110 and a handheld controller 112. Also shown is a user 105 wearing the HMD 110 and holding the handheld controller 112.

[0030] Using the examples disclosed herein, user 105 can align HMD 110 and controller 112 with each other. Alignment can be performed, for example, during an initialization process or to correct for drift between HMD 110 and controller 112 over time. By aligning HMD 110 and controller 112 with each other, the representation of controller 112's positioning within the VR environment of HMD 110 is aligned with the actual position of controller 112 relative to HMD 110. Additionally, the alignment of one or both of HMD 110 and controller 112 may drift relative to the VR scene. The embodiments disclosed herein can be used to realign one or both of HMD 110 and controller 112 to the VR scene.

[0031] In some embodiments, HMD 110 and controller 112 each independently (e.g., using separate inertial motion units) measure changes in movement and / or rotation. Using these measurements, HMD 110 and controller 112 each independently track their own orientation and / or attitude. In some embodiments, attitude includes position and orientation. For example, position can be represented by three degrees of freedom (3DOF) including forward / backward (X-axis), right / left (Y-axis), and up / down (Z-axis), and orientation can include three rotational components, such as pitch (i.e., rotation about the Y-axis), yaw (i.e., rotation about the Z-axis), and roll (i.e., rotation about the X-axis). Thus, in some embodiments, attitude is represented by six degrees of freedom (6DOF) including forward / backward (X-axis), right / left (Y-axis), up / down (Z-axis), pitch (i.e., rotation about the Y-axis), yaw (i.e., rotation about the Z-axis), and roll (i.e., rotation about the X-axis). While many examples are described herein regarding the alignment of the poses of HMD 110 and controller 112, other embodiments are possible. For example, some embodiments align one or the other in the position or orientation of HMD 110 and controller 112. Embodiments of the techniques described herein align controller 112 and / or HMD 110 with each other and / or with the VR scene.

[0032] Over time, as HMD 110 and controller 112 independently track their poses, numerical deviations can accumulate, causing alignment drift between HMD 110 and controller 112. For example, after a certain period of time, the pose of controller 112 relative to HMD 110 in real space may differ from the pose of the virtual representation of controller 112 relative to the virtual representation of HMD 110 in the VR environment.

[0033] Therefore, user 105 may perceive that they are pointing controller 112 in a specific direction relative to what they are looking in that direction (e.g., looking straight ahead with their eyes, and / or by pointing HMD 110 straight ahead). However, in reality, user 105 may be looking and pointing in a slightly different direction. This effect is called drift and can cause some difficulties for user 105 when interacting with virtual objects in the VR environment. Therefore, examples are disclosed herein that enable users to easily align HMD 110 with controller 112, for example, during VR application startup and / or later during operation. The alignment process adjusts the pose of controller 112 in HMD 110 to align with the pose of HMD 110. In some examples, alignment with HMD 110 and controller 112 is achieved via, for example, short-range wireless signals such as... The communication coupling occurs together. In this way, a single actuation of an input device (e.g., pressing a button to an active state) can trigger the communication coupling and alignment between the HMD 110 and the controller 112. In some embodiments, the alignment process is also performed after the communication coupling. For example, in some embodiments, the alignment process is performed multiple times during a VR session for performing initial alignment and / or then later for realignment. Although for simplicity, reference is made herein to buttons and button presses, any other input device can be used to trigger the communication coupling and alignment according to this disclosure. For example, a touch or gesture on a trackpad or any other detectable input.

[0034] An example alignment process includes: detecting a press of a button on a first component (e.g., handheld controller 112); communicatively coupling controller 112 to a second component (e.g., HMD 110) in response to the button press; and displaying a first symbol (e.g., target 114) associated with HMD 110 and a second symbol (e.g., cursor 115) associated with controller 112 in the VR scene in response to the communicative coupling. In some examples, cursor 115 may be represented as a laser pointer or a ray of light associated with (or emitted from) the handheld controller. The user is instructed to orient the HMD and / or their eyes toward the first symbol and to point the handheld controller 112 toward the first symbol. Additionally, data indicating alignment of controller 112 with HMD 110 (also referred to as alignment data) is stored when target 114 and cursor 115 overlap in the VR scene. In some examples, the VR scene changes simultaneously with target 114 and cursor 115 being displayed in a changing VR scene.

[0035] In some examples, alignment data represents a correction in at least one component of the pose of controller 112 relative to HMD 110. For example, when a user attempts to point both controller 112 and HMD 110 in a specified direction (e.g., directly forward, right, etc.); controller 112 may be pointed in a different direction than HMD 110. Therefore, example alignment data represents an offset to be applied to one or more components of a 3DoF or 6DoF reference model of controller 112 maintained by HMD 110 or another component of system 100. Alignment data can be applied by adding values ​​of alignment data to, for example, 3DoF or 6DoF coordinates, such that the measured pose is changed to a corrected pose. For example, alignment data could represent a difference between the poses of HMD and handheld controllers. Alignment data can be similarly applied between the HMD pose and the origin or a specified alignment point of the VR scene (see [link to relevant documentation]). Figure 9 ).

[0036] In some examples, it may be necessary to press and hold the button. In some cases, the roles of controller 112 and HMD 110 are reversed; for example, the button is pressed on HMD 110 instead of controller 112. Alternatively, it may be necessary to press and hold the button on both handheld controller 112 and HMD 110.

[0037] The attitude of HMD 110 can be determined using a transmitter or image (one of which is specified at reference numeral 116) and / or using a sensor or camera (one of which is specified at reference numeral 118) to determine the orientation of HMD 110 (e.g., in 6DoF or 3DoF). HMD 110 may include a camera 240 for sensing the transmitter / image 116 (see Figure 118). Figure 2 ), and / or for transmitter 245 sensed by sensor / camera 118 (see Figure 2 Any number and / or type of transmitter, image, sensor, and / or camera can be used, and any method utilizing these to determine the attitude of the HMD 110 can be used. The same applies to the transmitter 320 of the sensor / camera 118 and the handheld controller 112 (see [link]). Figure 3 The attitude of the handheld controller 112 is determined by the HMD 110 of the VR system 100 or another device (e.g., any of devices 131-133 discussed below), and / or can be determined by... Figure 12 Example computing devices P00 and P50 are implemented.

[0038] like Figure 1As shown, the example VR system 100 includes any number of computing and / or electronic devices that can exchange data via network 120. These devices may represent clients or servers and may communicate via network 120 or any other additional and / or alternative networks. Example client devices include, but are not limited to, mobile devices 131 (e.g., smartphones, personal digital assistants, portable media players, etc.), tablets (not shown), laptops or netbooks 132, cameras (not shown), HMDs 110, desktop computers 133, VR handheld controllers 112, gaming devices (not shown), and any other electronic or computing devices that can use network 120 or other networks to communicate with other computing or electronic devices or systems or can be used to access VR content or operate in a VR environment. Devices 110, 112, and 131-133 may represent client or server devices. Devices 110, 112, and 131-133 may execute a client operating system and one or more client applications that may access, render, provide, or display VR content on a display device included in or in combination with each of the respective devices 110, 112, and 131-133.

[0039] In some examples, the mobile device 131 may be placed, positioned, or otherwise implemented within or in conjunction with the HMD 110 to provide a display that can be used as a screen for the HMD 110. The mobile device 131 may additionally or alternatively include hardware and / or software for performing VR applications.

[0040] Figure 2 It is especially suitable for implementation Figure 1 A schematic diagram of Example HMD 110 and Example HMD 200. Figure 2 The example HMD 200 includes an optical module 205, a processor module 210, and a sensing module 215.

[0041] Optical module 205 includes, but is not limited to, lenses, mirrors, coatings, apertures, etc., that enable the wearer of HMD 200 to view display 220. Example display 220 displays side-by-side images and / or video for the respective eyes of the wearer of HMD 200. In other examples, two (or more) displays 220 are used separately to jointly display side-by-side images and / or video.

[0042] In order to control the HMD 200, Figure 2Example processor module 210 includes processor 225 in the form of a microcontroller, central processing unit (CPU), and / or graphics processing unit (GPU) programmed or configured to execute machine-readable instructions stored in memory 230. In some examples, the instructions, when executed, cause processor 225, particularly for VR applications, to determine the pose of HMD 200 and / or controller 112, and / or to perform an alignment process to align HMD 200 with controller 112.

[0043] In some examples, alignment data and / or parameters are stored in memory 230. In some embodiments, alignment data represents the difference or offset between a first direction in which the HMD 200 is oriented and a second direction in which the handheld controller is oriented during an alignment process assumed to be maintained with the intention of aligning with the HMD 200. For example, alignment data may represent the offset between the controller's longitudinal axis and the direction relative to the front surface of the HMD 200 (e.g., a vector orthogonal to the surface of the HMD 200). Thus, example alignment data could indicate that the user unintentionally points the controller 112 1 degree to the right and 2 degrees up while pointing at the controller and looking at the same point in the VR scene. The HMD 200 or another device 131-133 performing the VR application could use the alignment data to correct the measured orientation of the handheld controller so that it points 1 degree to the left and 2 degrees down, so that the handheld controller is now more accurately pointing at what the user is looking at. In some examples, what the wearer is looking at is determined by the position of the HMD. In other examples, what the wearer is looking at is determined using eye tracking. Alignment data and / or parameters can also compensate for any drift of the target in the HMD and the handheld controller due to, for example, accumulated numerical deviations, mechanical changes over time, user fatigue, etc. In some embodiments, alignment data includes data that can be used to align the attitude of the controller and the HMD (e.g., offsets for both orientation and position).

[0044] In some embodiments, alignment data represents a reference controller coordinate system for the controller and / or a reference HMD coordinate system. For example, the reference controller coordinate system may store the X-axis aligned with the vertical axis of the controller when the user is aiming forward at the controller as indicated during the alignment process, while the reference HMD coordinate system may store the X-axis aligned with the direction in which the front surface of the HMD faces when the user is facing forward as indicated during the alignment process.

[0045] To communicatively couple the example HMD 200 to a handheld controller, such as Figure 1 Controller 112, Figure 2The example HMD 200 includes a communication module 235. The example communication module 235 is a short-range wireless communication module according to the Bluetooth standard. However, other communication signals and / or protocols, such as Wi-Fi or Universal Serial Bus (USB), can be used.

[0046] To help determine the orientation of HMD 200, Figure 2 Example sensing module 215 may include features that can be used with Figure 1 Example transmitter / image 116 is used to determine the orientation of HMD 200 using a forward sensor or camera 240. Additionally or alternatively, example sensing module 215 may include other devices of the VR system, such as example devices 131-133, that may use transmitter 245 to determine the orientation of HMD 200. Some embodiments include an inertial measurement unit (IMU) 260 that can be used to determine the orientation of HMD 200 and / or detect the motion of HMD 200. In various embodiments, IMU 260 includes various types of sensors, such as, for example, accelerometers, gyroscopes, magnetometers, and other such sensors. The position and orientation of HMD 200 can be detected and tracked based on data provided by the sensors included in IMU 260. The detected position and orientation of HMD 200 can allow the system to further detect and track the user's head gaze direction and movement.

[0047] To determine the direction the wearer is gazing, example processor module 210 includes eye-tracking module 250. Using any number and / or type of algorithms, methods, and logic, Figure 2 The processor module 210 processes one or more images from the wearer's eyes to determine the direction the wearer is gazing.

[0048] To enable a user to control or operate the HMD 200, the example processor module 210 includes an input device 255. Examples of input devices 255 include buttons and other user-actuable controls. The example input device 255 of the HMD 200 can be operated by the user via communication modules 235 and 315 (see...). Figure 3 Initiate communication coupling between HMD 200 and handheld controller (not shown), and initiate alignment between HMD 200 and handheld controller.

[0049] Modules 205, 210, and 215, and the elements therein, can be implemented using any number and / or type of components and / or machine-readable instructions. In some examples, processor module 210 is implemented using a mobile device such as a mobile phone or smartphone communicatively and / or physically coupled to HMD 200. Sensing module 215 may be additionally implemented by a mobile device.

[0050] Figure 3The diagram can be used to implement Figure 1 Example handheld controller 112 and example handheld controller 300. For control Figure 3 The handheld controller 300, for example, includes a processor 305, which is, for example, in the form of a CPU or microcontroller programmed or configured to execute machine-readable instructions stored in memory 310. In some examples, the instructions, when executed, cause the processor 305 to interact with HMDs 110 and 200, and to operate a communication module 315.

[0051] In order to communicatively couple the example handheld controller 300 to the HMD 110, 200, Figure 3 The example handheld controller 300 includes a communication module 315. The example communication module 315 is a short-range wireless communication module according to the Bluetooth standard. However, other communication signals and / or protocols, such as Wi-Fi, USB, etc., can be used.

[0052] To help determine the location of the handheld controller 300, Figure 3 The example handheld controller 300 may include other devices in the VR system, such as example devices 110, 200, and 131-133, which may use a transmitter 320 to determine the orientation of the handheld controller 300. In some examples, a reflector may be implemented to reflect light emitted by other devices to determine the orientation of the controller 300. Additionally or alternatively, some embodiments of the handheld controller 300 include an inertial measurement unit (IMU) 330 to track the orientation and / or motion of the handheld controller 300. The IMU 330 may be similar to the IMU 260 of the HMD 200.

[0053] To enable a user to control or operate the handheld controller 300, an example handheld controller 300 includes a user-operable input device 325. An example of the input device 325 is a button that the user can press to activate. The example input device 325 of the handheld controller 300 can be operated by the user to initiate communication coupling between the handheld controller 300 and the HMD via communication modules 235 and 315, and to initiate alignment between the handheld controller 300 and the HMD.

[0054] To provide illumination for the sensor / camera 240 and / or other devices 131-133 that can determine the location of the handheld controller 300, the example handheld controller 300 includes a transmitter 320. The transmitter 320 may emit, for example, visible light and / or infrared light.

[0055] Figure 4 It can be, for example, used by one or more processors such as Figure 12The example processor executes machine-readable instructions to implement the flowchart of example procedure 400 of the example HMD disclosed herein. Example procedure 400 will be described with reference to example HMD 200, but other HMDs can be implemented using example procedure 400. Figure 4 Example method 400 includes processor 225 initiating VR mode on HMD 200 (block 405) and, for example, in Bluetooth discovery mode, initiating communication module 235 (block 410).

[0056] If the user actuates an input device such as input device 255 (e.g., by pressing and holding a button) (block 415), communication module 235 scans for nearby handheld controllers (block 420). If a recently connected handheld controller is found (block 425), communication module 235 attempts to connect to that handheld controller (block 430). In some examples, it is not necessary to actuate input device 255; instead, HMD 200 remains in listening mode when in VR mode, thus actively listening for Bluetooth queries from handheld controllers.

[0057] If communication is successfully established (block 435), the processor 225 displays the larger VR scene 620 on the display 220 (within the HMD) (see [link]). Figure 7 Part 615 (see) Figure 6 Initiating target 605 in ) (see Figure 6 The display of the VR scene 620, including the VR portion 615, the target 605, and the cursor 610, instructs the user to look at the target 605, causing the HMD and controller to be pointed in the same direction, and directing the connected handheld controller towards the target 605 (block 440). The VR portion 615 of the VR scene 620, the target 605, and the cursor 610 are displayed in the HMD. When the cursor 610 (see...) Figure 6When aligned (e.g., overlapped) with target 605 under the control of the handheld controller, input device 255 (if actuated) or input device 325 (if actuated) is released to indicate that target 605 and cursor 610 are aligned (block 445), the position and orientation of HMD and handheld controller are stored as alignment parameters in, for example, memory 230 (block 450), and the VR application is launched (block 455). In some examples, the alignment data represents a correction in at least one orientation component of controller 112 relative to HMD 110. For example, when the user instructs both controller and HMD to be pointed in the same direction, controller 112 may actually be pointed in a different direction than HMD 110. Thus, example alignment data may include data representing the offset of one or more components (e.g., in any component of a 3DoF or 6DoF reference coordinate system) of the attitude to be applied to controller 112 or HMD 110. Alignment parameters may be determined, for example, using transmitters 320 and 245, sensor / camera 240, and / or eye-tracking module 250. Control exits from example method 400 until a new communication coupling and / or alignment is required, which can be triggered by a user request, the expiration of a time period, an event occurring in the VR application, or any other type of event. Alignment data, such as alignment parameters, is then applied when determining the position and / or orientation of the handheld controller, which can then be used to determine the target location (or object) to which the connected handheld controller is being pointed. Due to the application of alignment data, the determined position and orientation can be determined more accurately. Therefore, in some implementations, the target location (or object) is also determined more accurately based on the application of alignment data. In other words, the representation of the handheld controller in the VR environment can be more accurately pointed at the user's intended target by using alignment data to correct for previous deviations or drifts when determining the orientation and position of the handheld controller.

[0058] Returning to block 435, if a connection cannot be established, control exits from example method 400 until a new attempt to communicatively couple and / or align the HMD to the handheld controller is initiated.

[0059] Returning to block 425, if the previously connected handheld controller is not found, communication module 235 searches for nearby handheld controllers (block 460). If no nearby handheld controller is found (block 460), control exits from exemplary method 400 until a new attempt to communicatively couple and / or align the HMD to a handheld controller is initiated. Otherwise, communication module 235 attempts to connect to the nearest handheld controller (block 465), and control proceeds to block 435.

[0060] Figure 7 And similar Figure 11These images are shown from the perspective of a third person viewing the VR environment. The people depicted in these images are within this VR environment with the third person present, and as seen by that third person.

[0061] Figure 5 It can be implemented, for example, by one or more processors such as Figure 12 The flowchart illustrates the example process 500 of the example handheld controller disclosed herein, executed by an example processor using machine-readable instructions. Example method 500 will be described with reference to example handheld controller 300, but other handheld controllers can be implemented using example method 500. Figure 5 Example method 500 includes waiting for user actuation and holding example input device 325 (block 505). When input device 325 is actuated and held (block 505), communication module 315 is activated in, for example, Bluetooth discovery mode (block 510), and communication module 315 scans for nearby HMDs (block 515). If a recently connected HMD is found (block 520), communication module 315 attempts to connect to that HMD (block 525).

[0062] If a connection is established (block 530), the processor 305 waits for the actuated input device 325 to be released (block 535). When it is determined that the input device 325 has been released (e.g., a button has been released), the processor 305 notifies the HMD 110 or other VR application implementing the alignment (block 540) that control exits from method 500 until a new attempt to communicatively couple and / or align the controller to the HMD is initiated.

[0063] Returning to block 530, if no connection is established (block 530), control exits from exemplary method 500 until a new attempt to communicatively couple and / or align the controller to the HMD is initiated.

[0064] Returning to block 520, if the previously connected HMD is not found, communication module 315 searches for nearby HMDs (block 545). If no nearby HMD is found (block 545), control exits from example method 500 until a new attempt to communicatively couple and / or align the controller to the HMD is initiated. Otherwise, the controller attempts to connect to the nearest HMD, and control proceeds to block 530.

[0065] Turning Figure 7 It shows including Figure 6 The VR scene 620 is part 615. As shown, the target 605 and the cursor 610 are overlaid on the VR scene 620, and if the user moves their gaze or body, the target 605 and the cursor 610 will be displayed on different parts of the VR scene 600.

[0066] Figure 8A and Figure 8B The diagram sequentially illustrates an example alignment of the HMD with the VR main screen (or other VR scene elements, such as objects). Figure 8A The illustration shows an example VR scene 805 in an HMD such as the HMD 110. The example VR scene 800 includes a main screen 810, which includes multiple buttons, one of which is designated at reference numeral 815. Due to drift, the center line 820 of the main screen 810 is offset relative to the center line 825 of the VR scene 805 (which is also the center line of the HMD). To align the HMD with the main screen 810, the user presses and holds a button on a handheld controller. When instructed, the HMD wearer looks (e.g., with their eyes and / or through the HMD pointing at them) at the center 830 of the main screen 810 and releases the button. Based on alignment data captured by the HMD, the main screen 810 is quickly returned to the center of the VR scene (and the HMD), as shown... Figure 8B As shown. In Figure 8A and Figure 8B In the example, only side-by-side offsets are depicted for clarity. Other offsets can be corrected, including offsets in one or more coordinates of a 3DoF or 6DoF reference module, for example. Such offsets can produce rotations, distortions, etc.

[0067] Figure 9 It can be, for example, used by one or more processors (e.g. Figure 12 The example processor (executable by the example) implements machine-readable instructions to implement the flowchart of example procedure 900 of the example HMD disclosed herein. Example method 800 will be described with reference to example HMD 200, but other HMDs can be implemented using example method 800. The example method 900 can be used as described above regarding... Figure 8A and Figure 8B The example alignment of the HMD and VR scene is performed as discussed. Part of example method 900 is similar to... Figure 4 Example method 400. Therefore, it will not be repeated here. Figure 4 and Figure 9 The same description of the same parts. Alternatively, interested readers can refer back to [the previous section]. Figure 4 Descriptions of the same parts.

[0068] At block 905, while wearing HMD 200, the user is instructed to look at the center of VR elements such as VR objects, the main screen, etc., and then stop actuating input device 255 (e.g., by releasing a button). When it is determined that input device 255 is no longer actuated (block 445), alignment data representing the offset between the direction HMD 110 is pointing and the positioning of the VR element's centerline is recorded and stored (block 450). At block 455, the VR application is launched and uses the stored alignment data to correct its position. For example, as... Figure 8A and Figure 8B As shown, the main screen, consisting of the button set, is shifted to the left to align with the center of the VR screen as defined by the alignment of HMD 200 (block 910). Control then exits from example method 800.

[0069] Figure 10 It can be, for example, used by one or more processors such as Figure 12 The example processor executes machine-readable instructions to implement the flowchart of example procedure 1000 of the example HMD disclosed herein. Example method 1000 will be described with reference to example HMD 200, but other HMDs can be implemented using example method 1000. The example method of method 1000 can be used to align a handheld controller to the HMD. Parts of example method 1000 are similar to... Figure 4 Example method 400. Therefore, it will not be repeated here. Figure 4 and Figure 10 The same description of the same parts. Alternatively, interested readers can refer back to [the previous section]. Figure 4 Descriptions of the same parts.

[0070] In block 1005, user 1105 (see also...) Figure 11 ) is instructed to look in a specified direction (e.g., straight ahead, right, etc.) and simultaneously at the same location 1115 (see Figure 11 At block 445, controller 1110 is pointed in that direction, and then actuation of input device 255 is stopped (e.g., by releasing a pressed button). When it is determined that input device 255 is no longer actuated (block 445), alignment data representing the offset between the pose of HMD 200 and the pose of the handheld controller is stored (block 450). In some embodiments, the alignment data represents the offset between the orientation of HMD 200 and the orientation of the handheld controller. At block 455, the VR application is launched and the pose of the controllers relative to HMD 110 is changed based on the alignment data, so that they are aligned with each other.

[0071] Can be combined Figure 8A , Figure 8B , Figure 9 and Figure 10An example is to simultaneously align the HMD with the VR element (e.g., the home screen) and to align the handheld controller with the HMD. In this example, the user will be instructed to look towards and point at the center of the VR element.

[0072] exist Figure 4 , Figure 9 and Figure 10 In this example, the alignment parameters are shown as being saved after the button is released. However, in some examples, the alignment parameters may be temporarily stored while the HMD is waiting for the button to be released.

[0073] One or more of the components and interfaces disclosed herein may be copied, implemented in parallel, implemented individually, combined, divided, rearranged, omitted, eliminated, and / or implemented in any other way. Furthermore, any of the disclosed components and interfaces may be implemented by… Figure 12 Example processor platforms P00 and P50 and / or one or more circuits, programmable processors, fuses, application-specific integrated circuits (ASICs), programmable logic devices (PLDs), field-programmable logic devices (FPLDs), and / or field-programmable gate arrays (FPGAs) are used for implementation. Any of the components and interfaces disclosed herein can be implemented, for example, as machine-readable instructions executable by one or more processors. They can be used, configured, and / or programmed using, etc. Figure 12 The controllers and / or any other suitable processing means shown herein are used to execute and / or carry out the examples disclosed herein. For example, any of these interfaces and elements may be embodied in program code and / or machine-readable instructions stored on a tangible and / or non-transitory computer-readable medium, which may be provided by a processor, computer, and / or machine having a processor, such as those described below. Figure 12 Machine access discussed together. Machine-readable instructions include, for example, instructions that cause a processor, computer, and / or machine having a processor to perform one or more specific processes. The order of execution of the methods can be changed, and / or one or more of the described blocks and / or interactions can be altered, eliminated, subdivided, or combined. Additionally, they can be executed sequentially and / or in parallel by, for example, separate processing threads, processors, devices, discrete logic, circuits, etc.

[0074] The example methods disclosed herein can be implemented as machine-readable instructions executable by one or more processors. Processors, controllers, and / or other similar devices can be used, configured, and / or programmed. Figure 12Any other suitable processing apparatus shown may be used to execute and / or implement the example methods. For example, they may be embodied in program code and / or machine-readable instructions stored on a tangible and / or non-transitory computer-readable medium, which may be provided by a processor, computer, and / or other machine having a processor, such as those described below. Figure 12 Machine access discussed together. Machine-readable instructions include, for example, instructions that cause a processor, computer, and / or machine having a processor to perform one or more specific processes. Many other methods may be employed to implement the example methods. For example, the order of execution may be changed, and / or one or more of the described blocks and / or interactions may be altered, eliminated, subdivided, or combined. Additionally, any or all of the example methods may be executed sequentially and / or in parallel by, for example, separate processing threads, processors, devices, discrete logic, circuits, etc.

[0075] As used herein, the terms “computer-readable medium” and “machine-readable medium” are explicitly defined to include any type of readable medium and explicitly exclude propagating signals. Examples of computer-readable and machine-readable media include, but are not limited to, one or any combination of the following: volatile and / or non-volatile memory, volatile and / or non-volatile memory devices, compact discs (CDs), digital universal discs (DVDs), read-only memory (ROM), random access memory (RAM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable PROM (EEPROM), optical storage discs, optical storage devices, magnetic storage discs, magnetic storage devices, caches, and / or any other storage medium in which information is stored for any duration (e.g., a prolonged period of time, permanently, temporarily, for temporary buffering and / or for caching information) and can be accessed by a processor, a computer and / or other machine having a processor.

[0076] Return to Figure 1 VR system 100 may include any number of VR content systems 140 storing content and / or VR software modules (e.g., VR application 144) capable of generating, modifying, and / or executing VR scenes. In some examples, devices 110, 112, and 131-133, as well as VR content system 140, include one or more processors and one or more memory devices capable of executing a client operating system and one or more client applications. HMD 110, controller 112, other devices 131-133, or VR content system 140 may be provided by Figure 12 Example computing devices P00 and P50 are implemented.

[0077] VR application 144 can be configured to run on any one or all of devices 110, 112, and 131-133. For example, HMD 110 can be connected to devices 131-133 to access VR content on VR content system 140. Devices 131-133 can be connected (wired or wirelessly) to HMD 110, which can provide VR content for display. The VR system can be just HMD 110, or a combination of devices 131-133 and HMD 110.

[0078] HMD 110 can refer to a VR HMD, glasses, eyepieces, or any other wearable device capable of displaying VR content. In operation, HMD 110 can execute VR application 144 that can play back received, rendered, and / or processed images for the user. In some cases, VR application 144 may be hosted by one or more of devices 131-133.

[0079] In some implementations, one or more content servers (e.g., VR content system 140) and one or more computer-readable storage devices may use network 120 to communicate with computing devices 110 and 131-134 to provide VR content to devices 110 and 131-134.

[0080] In some embodiments, mobile device 131 may execute VR application 144 and provide content for a VR environment. In some embodiments, laptop computing device 132 may execute VR application 144 and provide content from one or more content servers (e.g., VR content server 140). One or more content servers and one or more computer-readable storage devices may use network 120 to communicate with mobile device 131 and / or laptop computing device 132 to provide content displayed in HMD 106.

[0081] When HMD 106 is wirelessly coupled to device 102 or device 104, the coupling may include the use of any wireless communication protocol. A non-exhaustive list of wireless communication protocols that may be used individually or in combination includes, but is not limited to, those from the Institute of Electrical and Electronics Engineers (IEEE). The standards family includes Wi-Fi or Wireless Local Area Network (WLAN), Bluetooth, Transmission Control Protocol / Internet Protocol (TCP / IP), satellite data network, cellular data network, Wi-Fi hotspot, Internet, and Wireless Wide Area Network (WWAN).

[0082] When the HMD 106 is electrically coupled to device 102 or 104, a cable with a suitable connector at either end for insertion into device 102 or 104 can be used. A non-exhaustive list of wired communication protocols that can be used individually or in combination includes, but is not limited to, IEEE 802.3x (Ethernet), power line networks, the Internet, coaxial cable data networks, fiber optic data networks, broadband or dial-up modem networks, and private communication networks (e.g., private local area networks (LANs), leased lines, etc.).

[0083] The cable may include Universal Serial Bus (USB) connectors at both ends. The USB connectors may be the same type of USB connector or may each be a different type of USB connector. Various types of USB connectors may include, but are not limited to, USB Type-A connectors, USB Type-B connectors, micro-USB A connectors, micro-USB B connectors, micro-USB AB connectors, USB 5-pin Mini-B connectors, USB 4-pin Mini-B connectors, USB 3.0 Type-A connectors, USB 3.0 Type-B connectors, USB 3.0 Micro B connectors, and USB Type-C connectors. Similarly, electrical coupling may include a cable with a suitable connector at either end for insertion into HMD 106 and device 102 or device 104. For example, the cable may include USB connectors at both ends. The USB connectors may be the same type of USB connector or may be different types of USB connectors. Any end of the cable used to couple device 102 or 104 to HMD 106 may be securely connected to device 102 or 104 and / or HMD 106.

[0084] Figure 12 Examples of general-purpose computer devices P00 and P50 that can be used with the technologies described herein are shown. Computer device P00 is intended to represent various forms of digital computers, such as laptop computers, desktop computers, tablet computers, workstations, personal digital assistants, televisions, servers, blade servers, mainframes, and other suitable computing devices. Computer device P50 is intended to represent various forms of mobile devices, such as personal digital assistants, cellular phones, smartphones, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are intended to be illustrative only and are not intended to limit the implementation of the inventions described and / or claimed in this document.

[0085] The computing device P00 includes a processor P02, a memory P04, a storage device P06, a high-speed interface P08 connected to the memory P04 and a high-speed expansion port P10, and a low-speed interface P12 connected to a low-speed bus P14 and a storage device P06. The processor P02 may be a semiconductor-based processor. The memory P04 may be a semiconductor-based memory. Each of the components P02, P04, P06, P08, P10, and P12 is interconnected using various buses and may be mounted on a common motherboard or otherwise, as appropriate. The processor P02 can process instructions executed within the computing device P00, including instructions stored in the memory P04 or on the storage device P06 to display graphical information for a GUI on an external input / output device such as a display P16 coupled to the high-speed interface P08. In other embodiments, multiple processors and / or multiple buses may be used in conjunction with multiple memories and multiple types of memories, as appropriate. Additionally, multiple computing devices P00 can be connected, each providing a portion of the necessary operation (e.g., as a server group, a set of blade servers, or a multiprocessor system).

[0086] Memory P04 stores information within computing device P00. In one embodiment, memory P04 is one or more volatile memory cells. In another embodiment, memory P04 is one or more non-volatile memory cells. Memory P04 may also be another form of computer-readable medium, such as a magnetic disk or optical disk.

[0087] Storage device P06 provides large-capacity storage for computing device P00. In one embodiment, storage device P06 may be or contain computer-readable media, such as floppy disk devices, hard disk devices, optical disk devices, magnetic tape devices, flash memory, or other similar solid-state storage devices or arrays of devices, including devices in storage area networks or other configurations. The computer program product may be tangibly embodied in an information carrier. The computer program product may also contain instructions that, when executed, perform one or more methods, such as those described above. The information carrier is a computer or machine-readable medium, such as memory P04, storage device P06, or memory on processor P02.

[0088] High-speed controller P08 manages bandwidth-intensive operations of computing device P00, while low-speed controller P12 manages lower bandwidth-intensive operations. This allocation of functions is merely exemplary. In one embodiment, high-speed controller P08 is coupled to memory P04, display P16 (e.g., via a graphics processor or accelerator), and high-speed expansion port P10, which can accept various expansion cards (not shown). In this embodiment, low-speed controller P12 is coupled to storage device P06 and low-speed expansion port P14. The low-speed expansion port, which may include various communication ports (e.g., USB, Bluetooth, Ethernet, Wireless Ethernet), can be coupled to one or more input / output devices, such as keyboards, pointing devices, scanners, or network devices such as switches or routers, for example, via a network adapter.

[0089] As shown in the figure, the computing device P00 can be implemented in many different forms. For example, it can be implemented as a standard server P20, or multiple times in a group of such servers. It can also be implemented as part of a rack server system P24. Furthermore, it can be implemented in a personal computer such as a laptop computer P22. Alternatively, components from the computing device P00 can be combined with other components in a mobile device (not shown), such as device P50. Each of such devices can contain one or more of the computing devices P00 and P50, and the entire system can consist of multiple computing devices P00 and P50 communicating with each other.

[0090] The computing device P50 includes a processor P52, a memory P64, input / output devices such as a display P54, a communication interface P66 and a transceiver P68, and other components. The device P50 may also be provided with storage devices, such as microdrives or other devices, to provide additional storage. Each of components P50, P52, P64, P54, P66, and P68 is interconnected using various buses, and some components may be mounted on a common motherboard or otherwise, as appropriate.

[0091] Processor P52 executes instructions within computing device P50, including instructions stored in memory P64. The processor can be implemented as a chipset comprising individual and multiple analog and digital processors. The processor can provide, for example, coordination for other components of device P50, such as control of user interfaces, applications running on device P50, and wireless communications performed by device P50.

[0092] Processor P52 can communicate with the user via control interface P58 and display interface P56 coupled to display P54. Display P54 can be, for example, a TFT LCD (Thin Film Transistor Liquid Crystal Display) or OLED (Organic Light Emitting Diode) display or other suitable display technology. Display interface P56 can include suitable circuitry for driving display P54 to present graphics and other information to the user. Control interface P58 can receive commands from the user and translate them for submission to processor P52. Furthermore, an external interface P62 can be provided to communicate with processor P52, enabling near-field communication between device P50 and other devices. External interface P62 may be provided in some embodiments for wired communication, or in other embodiments for wireless communication, and multiple interfaces may be used.

[0093] Memory P64 stores information within computing device P50. Memory P64 may be implemented as one or more computer-readable media, one or more volatile memory cells, or one or more non-volatile memory cells. Extended memory P74 may also be provided and connected to device P50 via an extended interface P72, which may include, for example, a SIMM (Single In-line Memory Module) card interface. This extended memory P74 may provide additional storage space for device P50, or it may store applications or other information for device P50. Specifically, extended memory P74 may include instructions for performing or supplementing the above-described processes, and may also include security information. Therefore, for example, extended memory P74 may be provided as a security module for device P50 and may be programmed with instructions that allow secure use of device P50. Furthermore, secure applications may be provided via a SIMM card along with additional information, such as placing identification information on the SIMM card in an unbreakable manner.

[0094] As discussed below, memory may include, for example, flash memory and / or NVRAM memory. In one embodiment, the computer program product is tangibly embodied in an information carrier. The computer program product contains instructions that, when executed, perform one or more methods, such as those described above. The information carrier is a computer or machine-readable medium that can be received, for example, via transceiver P68 or external interface P62, such as memory P64, extended memory P74, or memory on processor P52.

[0095] Device P50 can communicate wirelessly via communication interface P66, which may include digital signal processing circuitry if necessary. Communication interface P66 can provide communication under various modes or protocols such as GSM voice calls, SMS, EMS or MMS messaging, CDMA, TDMA, PDC, WCDMA, CDMA2000, or GPRS. This communication can occur, for example, via radio frequency transceiver P68. Furthermore, short-range communication can occur using transceivers such as Bluetooth, Wi-Fi, or others (not shown). Additionally, a GPS (Global Positioning System) receiver module P70 can provide additional navigation and positioning-related wireless data to device P50, which can be used as appropriate by applications running on device P50.

[0096] Device P50 can also use audio codec P60 for audible communication, which receives spoken information from the user and converts it into usable digital information. Audio codec P60 can also generate audible sound for the user, such as through a speaker, for example, in the earpiece of device P50. This sound can include sounds from voice telephone calls, recorded sounds (e.g., voice messages, music files, etc.), and sounds generated by applications operating on device P50.

[0097] As shown in the figure, the computing device P50 can be implemented in many different forms. For example, it can be implemented as a cellular phone P80. It can also be implemented as part of a smartphone P82, a personal digital assistant, or other similar mobile devices.

[0098] Various implementations of the systems and techniques described herein can be implemented using digital electronic circuits, integrated circuits, specially designed ASICs (Application-Specific Integrated Circuits), computer hardware, firmware, software, and / or combinations thereof. These various implementations may include implementations in one or more computer programs executable and / or interpretable on a programmable system, said programmable system including at least one programmable processor, which may be dedicated or general-purpose, coupled to receive data and instructions from a storage system, at least one input device, and at least one output device, and to transmit data and instructions to the storage system, at least one input device, and at least one output device.

[0099] These computer programs (also referred to as programs, software, software applications, or code) include machine instructions for a programmable processor and can be implemented using high-level procedural and / or object-oriented programming languages ​​and / or assembly / machine languages. As used herein, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, device, and / or apparatus (e.g., disk, optical disk, memory, programmable logic device (PLD)) used to provide machine instructions and / or data to a programmable processor, including machine-readable media that receive machine instructions as machine-readable signals. The term "machine-readable signal" refers to any signal used to provide machine instructions and / or data to a programmable processor.

[0100] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user and a keyboard and pointing device (e.g., a mouse or trackball) that the user can use to provide input to the computer. Other types of devices may also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form, including sound, speech, or tactile input.

[0101] The systems and technologies described herein can be implemented in computing systems that include back-end components (e.g., as data servers), or middleware components (e.g., application servers), or front-end components (e.g., client computers having a graphical user interface or web browser that a user can use to interact with implementations of the systems and technologies described herein), or any combination of such back-end, middleware, or front-end components. Components of the system can be interconnected via digital data communication (e.g., communication networks) of any form or medium. Examples of communication networks include local area networks (“LANs”), wide area networks (“WANs”), and the Internet.

[0102] A computing system may include clients and servers. Clients and servers are generally geographically separated and typically interact via a communication network. The client-server relationship is established by computer programs running on the respective computers and having a client-server relationship with each other.

[0103] In some implementations... Figure 12 The computing device depicted may include sensors that interface with a virtual reality (VR headset / HMD device P90) to generate an immersive environment. For example, including... Figure 12One or more sensors on the computing device P50 or other computing device depicted herein may provide input to the VR headset P90, or more generally, to the VR space. Sensors may include, but are not limited to, touchscreens, accelerometers, gyroscopes, pressure sensors, biometric sensors, temperature sensors, humidity sensors, and ambient light sensors. The computing device P50 may use the sensors to determine the absolute position of the computing device in the VR space and / or detected rotation, which can then be used as input to the VR space (e.g., to perform alignment as described herein or to interact with the VR space after alignment has been performed). For example, the computing device P50 may be incorporated into the VR space as a virtual object such as a controller, laser pointer, keyboard, weapon, etc. User positioning of the computing device / virtual object when incorporated into the VR space allows the user to position the computing device to view the virtual object in some way within the VR space. For example, if the virtual object represents a laser pointer, the user can manipulate the computing device as if it were an actual laser pointer. The user can move the computing device left and right, up and down, in a circle, etc., and use the device in a manner similar to using a laser pointer.

[0104] In some implementations, one or more input devices included on or connected to the computing device P50 can be used as input to the VR space. Input devices may include, but are not limited to, touchscreens, keyboards, one or more buttons, trackpads, touchpads, pointing devices, mice, trackballs, joysticks, cameras, microphones, headphones or earphones with input functionality, game controllers, or other connectable input devices. User interaction with the input devices included on the computing device P50 when the computing device is incorporated into the VR space can result in specific actions occurring within the VR space.

[0105] In some implementations, the touchscreen of the computing device P50 can be rendered as a touchpad in VR space. The user can interact with the touchscreen of the computing device P50. For example, in a VR headset P90, the interaction is rendered as movement on a touchpad rendered in VR space. The rendered movement can control virtual objects in VR space.

[0106] In some implementations, one or more output devices included on the computing device P50 may provide output and / or feedback to a user of the VR headset P90 in the VR space. The output and feedback may be visual, tactile, or audio. The output and / or feedback may include, but is not limited to, vibration, turning one or more lights or flashes on and off or blinking and / or flashing, alarm sounds, ringtones, music playback, and playback of audio files. Output devices may include, but are not limited to, vibration motors, vibration coils, piezoelectric devices, electrostatic devices, light-emitting diodes (LEDs), flashlights, and speakers.

[0107] In some implementations, the computing device P50 can appear as another object in a computer-generated 3D environment. Interactions between the user and the computing device P50 (e.g., rotation, shaking, touching the touchscreen, swiping a finger across the touchscreen) can be interpreted as interactions with objects in VR space. In the example of a laser pointer in VR space, the computing device P50 appears as a virtual laser pointer in the computer-generated 3D environment. When the user manipulates the computing device P50, the user in VR space sees the laser pointer move. The user receives feedback from interactions with the computing device P50 in the VR environment, either on the computing device P50 or on the VR headset P90. The alignment techniques described herein can allow these interactions to feel more natural and intuitive to the user because the position of the corresponding object rendered in the VR headset P90 will be aligned with the actual object.

[0108] In some implementations, the computing device P50 may include a touchscreen. For example, a user can interact with the touchscreen in a specific way that mimics what happens on the touchscreen, based on what is happening in VR space. For instance, a user can use pinch-like movements to zoom the content displayed on the touchscreen. Such pinch movements on the touchscreen allow information provided in VR space to be zoomed. In another example, the computing device may be rendered as a virtual book in a computer-generated 3D environment. In VR space, pages of the book can be displayed, and a user's finger swiping across the touchscreen can be interpreted as turning / flipping the pages of the virtual book. With each page turned / flipped, in addition to seeing the page content change, the user can also be provided with audio feedback, such as the sound of turning pages in a book.

[0109] In some implementations, one or more input devices (e.g., mouse, keyboard) other than a computing device may be rendered in a computer-generated 3D environment. The rendered input devices (e.g., rendered mouse, rendered keyboard) can be used to control objects in VR space as if they were rendered in VR space.

[0110] In this specification and the appended claims, the singular forms “a,” “an,” and “the” do not exclude plural references unless the context clearly specifies otherwise. Additionally, conjunctions such as “and,” “or,” and “and / or” are included unless the context clearly specifies otherwise. For example, “A and / or B” includes only A, only B, and A together with B. Furthermore, the connecting lines or connectors shown in the various figures are intended to represent exemplary functional relationships and / or physical or logical couplings between various elements. It should be noted that many alternative or additional functional relationships, physical connections, or logical connections may exist in actual devices. Moreover, unless an element is specifically described as “essential” or “critical,” an item or component is not essential to the practice of the embodiments disclosed herein.

[0111] The following examples illustrate alternative implementation methods:

[0112] Example 1: A method comprising: detecting a first input from a handheld controller of a virtual reality system; in response to detecting the first input, instructing a user to orient the handheld controller in a specified direction; detecting a second input from the handheld controller; and in response to detecting the second input, storing alignment data representing the alignment of the handheld controller.

[0113] Example 2: The method according to Example 1 further includes: in response to detecting the first input, communicatively coupling the handheld controller to the head-mounted display of the virtual reality system.

[0114] Example 3: The method according to Example 2 further includes: in response to detecting the second input: determining the controller orientation of the handheld controller; and determining the display orientation of the head-mounted display (HMD).

[0115] Example 4: According to the method of Example 3, storing alignment data includes storing data representing the offset between the determined controller orientation and the determined display orientation.

[0116] Example 5: According to the method of Example 2, the communicative coupling includes establishing a connection via a short-range wireless signal.

[0117] Example 6: The method according to any one of Examples 1 to 5 further includes aligning at least a portion of the virtual reality scene displayed by the head-mounted display based on the alignment data.

[0118] Example 7: The method according to one of Examples 1 to 6, wherein storing alignment data representing the alignment of the handheld controller includes storing at least one of forward / backward position, up / down position, left / right position, pitch, roll and yaw.

[0119] Example 8: The method according to one of Examples 1 to 7, wherein detecting the first input includes detecting actuation of the input device of the handheld controller.

[0120] Example 9: According to the method of Example 8, wherein detecting the actuation of the input device includes detecting the pressing of a button on the handheld controller.

[0121] Example 10: The method according to Example 8 or 9, wherein detecting the second input includes detecting the release of the input device of the handheld controller.

[0122] Example 11: According to the method of Example 10, the release of the input device includes the release of the button.

[0123] Example 12: A computer program product including a non-transitory storage medium, the computer program product including code that, when executed by processing circuitry of a virtual reality system configured to generate a virtual reality environment, causes the processing circuitry to perform a method, the method comprising: detecting a first input from a handheld controller of the virtual reality system; communicatively coupling the handheld controller to a head-mounted display of the virtual reality system in response to detecting the first input; instructing a user to orient the handheld controller in a specified direction; detecting a second input from the handheld controller; and storing alignment data representing the alignment of the handheld controller in response to detecting the second input.

[0124] Example 13: The computer program product according to Example 12, wherein the method further includes: in response to detecting the second input: determining the controller orientation of the handheld controller; and determining the display orientation of the head-mounted display (HMD).

[0125] Example 14: The computer program product according to Example 13, wherein storing alignment data includes storing data representing the offset between the determined controller orientation and the determined display orientation.

[0126] Example 15: A computer program product according to one of Examples 12 to 14, wherein communicative coupling includes establishing a connection via a short-range wireless signal.

[0127] Example 16: A computer program product according to one of Examples 12 to 15 further includes aligning at least a portion of a virtual reality scene displayed by the head-mounted display based on the alignment data.

[0128] Example 17: A computer program product according to one of Examples 12 to 16, wherein instructing a user to orient a handheld controller in a specified direction includes displaying instructions within a virtual reality scene displayed by the head-mounted display.

[0129] Example 18: An electronic device configured to generate a virtual reality environment, the electronic device comprising: a memory; and control circuitry coupled to the memory, the control circuitry being configured to: detect a first input from a handheld controller of the virtual reality system; in response to detecting the first input: communicatively couple the handheld controller to a head-mounted display of the virtual reality system via a wireless signal; and instruct a user to orient the handheld controller in a specified direction; detect a second input from the handheld controller; and in response to detecting the second input: determine a controller orientation of the handheld controller; determine a display orientation of the head-mounted display; and store alignment data representing the alignment of the handheld controller.

[0130] Example 19: An electronic device according to Example 18, wherein storing alignment data includes storing data representing the offset between a determined controller orientation and a determined display orientation.

[0131] Example 20: In the electronic device according to Example 19, the control circuitry is further configured to coordinate the movement of the handheld controller and the head-mounted display in a virtual reality scene based on stored alignment data.

[0132] Although certain example methods, apparatuses, and articles have been described herein, the scope of this patent is not limited thereto. Rather, this patent covers all methods, apparatuses, and articles that substantially fall within the scope of the claims of this patent.

Claims

1. A method comprising: The user is instructed to orient the handheld controller in a specified direction by the following operation: displaying the target within the scene shown by the head-mounted display; as well as The user is instructed to look at the target, so that the head-mounted display and the handheld controller point in the same direction; If a connection is established between the head-mounted display and the handheld controller: Detect input from the handheld controller; and In response to the detection of the input: Determine the orientation of the handheld controller; The orientation associated with the head-mounted display is determined by the following steps: instructing the connected handheld controller to point at the target; when the cursor under the control of the handheld controller is aligned with the target, the input device of the handheld controller is released if actuated to indicate that the target and the cursor are aligned; as well as Alignment data representing the alignment of the handheld controller is stored, the alignment data including data representing the offset between the determined orientation of the handheld controller and the determined orientation associated with the head-mounted display.

2. The method of claim 1, further comprising aligning at least a portion of the scene displayed by the head-mounted display based on the alignment data.

3. The method of claim 1, further comprising determining the orientation of the handheld controller with respect to the scene displayed by the head-mounted display based on the alignment data.

4. The method according to claim 1, wherein, The alignment data representing the alignment of the handheld controller includes storing at least one of forward / backward position, up / down position, left / right position, pitch, roll, and yaw.

5. The method according to claim 1, wherein, The head-mounted display is communicatively coupled to the handheld controller.

6. A computer program product including a non-transitory storage medium, the computer program product including code, the code causing the processing circuitry to perform a method when executed by processing circuitry, the method comprising: The user is instructed to orient the handheld controller in a specified direction by displaying the target within a scene displayed by a head-mounted display communicatively coupled to the handheld controller; as well as The user is instructed to look at the target, so that the head-mounted display and the handheld controller point in the same direction; If a connection is established between the head-mounted display and the handheld controller: Detect input from the handheld controller; and In response to the detection of the input: Determine the orientation of the handheld controller; The orientation associated with the head-worn display, which is communicatively coupled to the handheld controller, is determined by the following steps: instructing the connected handheld controller to point at the target; when the cursor under the control of the handheld controller is aligned with the target, the input device of the handheld controller is released if actuated to indicate that the target and the cursor are aligned; as well as Alignment data representing the alignment of the handheld controller is stored, the alignment data including data representing the offset between the determined orientation of the handheld controller and the determined orientation associated with the head-mounted display.

7. The computer program product according to claim 6, wherein, Determining the orientation associated with a head-mounted display includes determining the orientation of the head-mounted display.

8. The computer program product according to claim 6, wherein, Determining the orientation associated with a head-mounted display includes determining the direction the user is gazing in.

9. The computer program product according to claim 6, wherein, The method also includes aligning at least a portion of the scene displayed by the head-mounted display based on the alignment data.

10. The computer program product of claim 6, the method further comprising determining, based on the alignment data, the orientation of the handheld controller with respect to a scene displayed by the head-mounted display.

11. The computer program product of claim 6, the method further comprising determining, based on the alignment data, an object within a scene displayed by the head-mounted display that the handheld controller is pointing at.

12. The computer program product according to claim 6, wherein, Instructions are displayed to instruct the user to orient the handheld controller in a specified direction within the scene displayed by the head-mounted display.

13. An electronic device, the electronic device comprising: Memory; as well as A control circuit coupled to the memory, the control circuit being configured to: Detect the first input from the handheld controller; In response to the detection of the first input: Display the target within a scene shown by a head-mounted display; as well as The user is instructed to look at the target, so that the head-mounted display and the handheld controller point in the same direction; If a connection is established between the head-mounted display and the handheld controller: Detecting a second input from the handheld controller; and In response to the detection of the second input: Determine the orientation of the handheld controller; The orientation associated with the head-mounted display is determined by the following steps: instructing the connected handheld controller to point at the target; when the cursor under the control of the handheld controller is aligned with the target, the input device of the handheld controller is released if actuated to indicate that the target and the cursor are aligned; as well as Alignment data representing the alignment of the handheld controller is stored, the alignment data including data representing the offset between the determined orientation of the handheld controller and the determined orientation associated with the head-mounted display.

14. The electronic device according to claim 13, wherein, Detecting the first input includes detecting the actuation of the input device of the handheld controller.

15. The electronic device according to claim 14, wherein, Detecting the actuation of the input device includes detecting the pressing of the buttons on the handheld controller.

16. The electronic device according to claim 15, wherein, Detecting the second input includes detecting the release of the input device of the handheld controller.

17. The electronic device according to claim 16, wherein, Detecting the release of the input device includes detecting the release of the button.

18. The electronic device of claim 13, wherein the control circuitry is further configured to coordinate the movement of the handheld controller and the head-mounted display in a scene based on stored alignment data.

Images