Mobile device-based VR control

Abstract

The present disclosure relates to controlling virtual reality (VR) content displayed on a VR head-mounted display (HMD). Communication can be established between a computer system, a VR HMD, and a mobile device. User input configured to control the VR content displayed on a display of the VR HMD can be received at the mobile device. The VR content displayed on the VR HMD can then be controlled based on the user input received at the mobile device.

Description

[Technical Field]

[0001] The present disclosure relates generally to the field of virtual reality, and more particularly to controlling virtual reality systems using mobile devices. [Background technology]

[0002] A virtual reality (VR) system uses computer technology to simulate a virtual environment. VR systems provide sensory data (e.g., audio and visual data) to users so that they experience an immersive environment. VR systems typically include a head-mounted display (HMD) that users wear to visualize the simulated environment. The simulated environment can be similar to the real world or completely fictional. Summary of the Invention

[0003] Embodiments of the present disclosure relate to a method, system, and computer program product for controlling virtual reality (VR) content displayed on a VR head-mounted display (HMD). Communication can be established between a computer system, a VR HMD, and a mobile device. User input configured to control the VR content displayed on a display of the VR HMD can be received at the mobile device. The VR content displayed on the VR HMD can then be controlled based on the user input received at the mobile device.

[0004] The above summary is not intended to describe each example embodiment or every implementation of the present disclosure. [Brief explanation of the drawings]

[0005] The drawings included in this disclosure are incorporated in and form a part of this specification. These drawings illustrate embodiments of the disclosure and, together with the description, explain the principles of the disclosure. The drawings are merely illustrative of typical embodiments and are not intended to limit the disclosure.

[0006] [Figure 1] FIG. 1 illustrates a VR environment in which exemplary embodiments of the present disclosure can be implemented. [Figure 2] FIG. 1 is a block diagram of an example computing environment in which exemplary embodiments of the present disclosure may be implemented. [Figure 3] 1 is a flowchart illustrating a process for controlling a virtual reality system using a mobile device according to an embodiment of the present disclosure. [Figure 4] FIG. 1 is an explanatory diagram illustrating various controls that can be used to control virtual reality content according to an embodiment of the present disclosure. [Figure 5] FIG. 1 illustrates a cloud computing environment according to an embodiment of the present disclosure. [Figure 6] FIG. 2 illustrates an abstraction model layer according to an embodiment of the present disclosure. [Figure 7] FIG. 1 is a schematic block diagram of an example computer system that can be used to implement one or more of the methods, tools, and modules described herein, and any associated functionality, according to embodiments of the present disclosure.

[0007] While the embodiments described herein are susceptible to various modifications and alternative forms, specific details of the embodiments are shown by way of example in the drawings and will be described in detail. However, the particular embodiments described should not be construed as limiting. Rather, it is intended to cover all modifications, equivalents, and alternatives falling within the scope of the present disclosure. DETAILED DESCRIPTION OF THE INVENTION

[0008] Aspects of the present disclosure relate generally to the field of virtual reality, and more particularly to controlling virtual reality systems using mobile devices. While the present disclosure is not necessarily limited to such applications, various aspects of the present disclosure can be understood through the description of various examples in this context.

[0009] A virtual reality (VR) system uses computer technology to simulate a virtual environment. VR systems typically include a head-mounted display (HMD), which provides sensory data, particularly visual data, to the user wearing the HMD. VR systems, such as the OCULUS RIFT® and HTC VIVE®, typically include controllers that users can hold in their hands to interact with the VR environment. The controllers act as the user's hands during VR use and allow the user to manipulate objects and views within the VR environment.

[0010] However, controllers associated with VR systems are typically expensive. Furthermore, some controllers may include unfamiliar control buttons and / or control technology, potentially hindering the user's VR experience. Furthermore, using dedicated VR controllers only adds to the hardware associated with a VR system, which may already include multiple sensors (for tracking), HMDs, computer systems, and the necessary cords and ports to interconnect related components. Therefore, there is a need for an alternative solution to dedicated VR controllers that provides intuitive functionality for controlling VR content.

[0011] Aspects of the present disclosure relate to a method for controlling virtual reality (VR) content displayed on a VR head-mounted display (HMD). Communication can be established between a computer system, a VR HMD, and a mobile device. User input configured to control the VR content displayed on a display of the VR HMD can be received at the mobile device. The VR content displayed on the VR HMD can then be controlled based on the user input received at the mobile device.

[0012] By allowing users to control VR content via mobile devices, it is possible to control the VR content using intuitive mobile device-based controls that users are already familiar with, instead of using dedicated VR controllers, which may be unfamiliar to users. Furthermore, there is no need to purchase additional hardware; users can control the VR system simply by connecting their usual mobile device to the VR system. Furthermore, the range of control options is expanded depending on the input available on the connected mobile device. For example, VR content can be controlled using in-air controls, bezel-based controls, touch controls, motion controls, and other controls, without being limited to the control mechanisms available with a dedicated VR controller.

[0013] Referring now to the drawings, Figure 1 is a diagram illustrating an example virtual reality (VR) environment 100 according to an embodiment of the present disclosure. The virtual reality environment 100 includes a VR head-mounted display (HMD) 105 communicatively coupled to a computer system 110. A VR user 115 is actively using the VR HMD 105 and controlling the VR experience (e.g., VR gameplay) using a smartphone 120 (e.g., controller 215-1 in Figure 2) and a smartwatch 125 (e.g., controller 215-2 in Figure 2).

[0014] Before using the VR HMD 105, the VR user 115 can set up the VR environment 100. Initially, the VR user 115 can install the necessary VR software on the computer system 110 and configure communication couplings between the computer system 110, the VR HMD 105, the first sensor 130, and the second sensor 135. After starting the VR software on the computer system 110, the VR user 115 can be prompted for a room setup. The room setup can ensure that the VR HMD is properly calibrated based on the characteristics of the VR user 115 (e.g., appropriate height and gaze direction) and that the VR user 115 does not bump into nearby objects. The room setup can include indicating the location of the ground, the user's height, and the direction the user 115 will face during VR use.

[0015] The space setup may further include defining a VR boundary 140. The VR boundary 140 may be defined to prevent the VR user 115 from bumping into stationary objects in the space. This may be accomplished by displaying the VR boundary 140 to the VR user 115 while using the VR HMD 105 to ensure that the VR user 115 stays within the VR boundary 140 and avoids bumping into nearby objects (e.g., nearby furniture, a television, or a counter). In an embodiment, the VR boundary 140 may only be displayed while the user is moving (e.g., to minimize distractions during immersion in the VR environment).

[0016] After configuring the VR environment 100, the VR user 115 can put on the VR HMD 105 and begin VR use (VR gameplay). The sensors 130 and 135 can be configured to track the VR user 115 during VR immersion. For example, the sensors 130 and 135 can track the VR HMD 105, the smartphone 120, and the smartwatch 125 to facilitate updates on the VR user's 115's position within the virtual reality. Additionally, the sensors 130 and 135 can utilize the tracking of the VR HMD 105, the smartphone 120, and the smartwatch 125 to determine the VR user's position relative to the VR boundary 140. This can alert the VR user 115 when they are approaching the VR boundary 140 to prevent collisions with nearby objects.

[0017] In embodiments, the smartphone 120 and smartwatch 125 can be used to control VR use (e.g., touch-based, bezel-based, motion-based, and air-based controls). Accordingly, the smartphone 120 and smartwatch 125 can operate to manipulate objects or viewing angles within VR gameplay. Examples of controls that can be performed using the smartphone 120 and smartwatch 125 include aiming (e.g., moving the user's camera view from the user's point of view), panning (e.g., realigning a top-down view of a virtual base map or moving the camera view along a horizontal plane), performing actions (e.g., interacting with objects), zooming, locomotion, and the like. Additionally, the smartphone 120 and smartwatch 125 can emulate a VR user's hands 125 (or virtual devices) within the virtual gameplay. For example, the smartphone 120 can be used to control the movement of the user's right hand, and the smartwatch 125 can be used to control the movement of the user's left hand.

[0018] It should be noted that FIG. 1 is intended to illustrate representative major components in an example VR environment 100. In some embodiments, individual components may be more complex or simpler than those depicted in FIG. 1. Furthermore, components other than or in addition to those depicted in FIG. 1 may be present, and the number, type, and configuration of such components may vary. Accordingly, system configurations may vary, and aspects of the present disclosure are not limited to any particular configuration described. For example, in an embodiment, VR environment 100 may include only smartphone 120 and not smartwatch 125. In an embodiment, the number of sensors may be greater or fewer. In an embodiment, additional display devices may be present in VR environment 100 (e.g., a monitor may mirror VR content displayed on VR HMD 105).

[0019] 2, a block diagram illustrating an example computing environment 200 in which exemplary embodiments of the present disclosure may be implemented. The computing environment 200 includes a VR HMD 205, a first controller 215-1, a second controller 215-2 (collectively referred to as controllers 215), at least one computer system 235, sensors 260, and a network 250.

[0020] The controller 215 and the computer system 235 each include one or more processors 220-1, 220-2 (collectively referred to as processors 220) and 240 and one or more memories 225-1, 225-2 (collectively referred to as memories 225) and 245. Additionally, the VR HMD 205 and the sensors 260 may each include one or more processors and memories (not shown). The controller 215 and the computer system 235 are configured to communicate with each other via internal or external network interfaces 230-1, 230-2 (collectively referred to as network interfaces 230) and 240. The network interfaces 210 and 240 are, in some embodiments, modems or network interface cards. Additionally, the VR HMD 205 and the sensors 260 may each include one or more network interfaces (not shown) to facilitate communication over the network 250. The controller 215 and / or the computer system 235 may include a display. Additionally, the controller 215 and / or the computer system 235 may include any configuration of input devices (e.g., keyboard, mouse, scanner, video camera, etc.), commercially available or custom software (e.g., browser software, communications software, server software, natural language processing software, search engine and / or web crawling software, VR software, etc.), or both.

[0021] In an embodiment, the controller 215 may be a mobile device used to control VR usage. Examples of mobile devices include smartphones, tablets, and wearable devices. The computer system 235 may be a VR-ready machine configured to load and execute VR programs, perform VR graphics rendering, and execute VR gameplay processing. In this manner, the processing power of the computer system 235 may be used to display VR content on the display 210 of the VR HMD 205. The sensors 260 may include a positioning system 265 configured to determine the location of components within the computing environment. For example, the positioning system 265 may be configured to determine the locations of the controller 215, the VR HMD 205, and the computer system 235 using indoor positioning system technologies (e.g., Wi-Fi-based positioning systems, Bluetooth-based positioning systems, angle of arrival, time of arrival, received signal strength indication (RSSI), etc.). By tracking the device relative to the computing environment 200, corresponding positioning adjustments can be simulated within the VR gameplay based on the physical movement of the device.

[0022] The controller 215, the VR HMD 205, the sensors 260 (collectively referred to as "devices"), and the computer system 235 may be remote from one another and communicate via the network 250. In some embodiments, the computer system 235 may be a central hub to which the devices can establish communication connections, such as in a client-server networking model. Alternatively, the computer system 235 and the devices may be configured in any other suitable networking relationship (e.g., using a peer-to-peer (P2P) configuration or any other network topology).

[0023] The network 250 can be implemented using any number of any suitable communication media. For example, the network 250 can be a wide area network (WAN), a local area network (LAN), the Internet, or an intranet. In particular embodiments, the devices and the computer system 235 are local to one another and can communicate via any suitable local communication medium. For example, the devices and the computer system 235 can communicate using a local area network (LAN), one or more wired connections, wireless links (e.g., Bluetooth®) or routers, or an intranet. In some embodiments, the devices and the computer system 235 can be communicatively coupled using one or more networks or one or more local connections, or a combination of both. For example, the VR HMD 205 can be wired to the computer system 235 (e.g., using an HDMI link or a USB link, or both), and the sensors 260 can communicate with the computer system 235 using a wireless link, such as Bluetooth.

[0024] In some embodiments, network 250 is implemented within a cloud computing environment or using one or more cloud computing services. Common to various embodiments, a cloud computing environment may include a network-based distributed data processing system that provides one or more cloud computing services. Furthermore, a cloud computing environment may include many computers (e.g., hundreds or thousands or more computers) located in one or more data centers and configured to share resources via network 250.

[0025] The first controller 215-1, the second controller 215-2, and the computer system 235 include VR applications 227-1, 227-2, and 247, respectively (collectively referred to as "VR applications"). The VR applications can be configured to provide various functionality to enable VR gameplay using devices in the system. The VR application 247 of the computer system 235 can be configured to load and run VR games / apps, monitor, pair, and update (e.g., firmware updates) the controllers 215-1, 215-2, the VR HMD 205, and the sensors 260, provide rules regarding the user's VR "play area" (e.g., sitting, standing, spatial scale, etc.), and facilitate user setting changes (e.g., graphics settings such as supersampling and motion smoothing, audio settings, layout settings).

[0026] VR applications 227-1 and 227-2 can be configured to provide functionality that enables controllers 215-1 and 215-2 to control VR gameplay. That is, controllers 215-1 and 215-2 can accept user input and send it to computer system 235 to control VR gameplay (displayed in VR HMD 205). In embodiments, VR application 247 provides updates to VR applications 227-1 and 227-2 so that devices emulating controllers 215-1 and 215-2 are configured to appropriately control VR use (e.g., so that a given input produces a desired output).

[0027] Examples of inputs that can be used to control VR usage include touch-based interactions (e.g., a user interacts with a touchscreen on a mobile device), tilt-based interactions (e.g., a user tilts a device to perform corresponding controls), inertial-based interactions (e.g., a user moves a device to perform corresponding controls), bezel-based interactions (e.g., a user interacts with a bezel portion of a mobile device; see bezel-based control 460 in FIG. 4), and air-to-air interactions (e.g., a user interacts in the air above a mobile device, which is captured by a camera or sensor (on the mobile device or another device) and converted into a corresponding output). Examples of outputs that can be obtained based on received inputs include performing an action in the virtual reality (e.g., interacting with an object in VR, such as picking up an item), aiming, moving, panning, zooming, motion, etc.

[0028] While Figure 2 illustrates a computing environment 200 having a single computer system 235, a suitable computing environment for implementing embodiments of the present disclosure may include any number of computer systems. The various models, modules, systems, and components illustrated in Figure 2, if present, may exist across multiple computer systems and devices. For example, some embodiments may include two computer systems. These two computer systems may be communicatively coupled by any suitable communication connection (e.g., by a WAN, LAN, wired connection, intranet, or the Internet).

[0029] It should be noted that FIG. 2 is intended to illustrate representative major components in an example computing environment 200. In some embodiments, individual components may be more complex or simpler than those depicted in FIG. 2. Furthermore, components other than or in addition to those depicted in FIG. 2 may be present, and the number, type, and configuration of such components may vary. Accordingly, system configurations may vary, and aspects of the present disclosure are not limited to any particular configuration described. For example, in some embodiments, computer system 235 may be integrated with VR HMD 205.

[0030] Next, FIG. 3 illustrates an example process 300 for controlling VR content displayed on a VR HMD using a mobile device, according to an embodiment of the present disclosure.

[0031] Process 300 begins with operation 305, in which communication is established between the computer system, the VR HMD, the sensor (optional), and the mobile device. Any of the communication technologies described with respect to network 250 of FIG. 2 can be used to interconnect the aforementioned components. In embodiments, one or more wired connections can be established between devices in the system. For example, an HDMI connection and / or a USB connection can be configured to interconnect the computer system to the VR HMD (e.g., to enable high-speed transfer of graphics data). In embodiments, wireless connections can be established between the computer system, the mobile device, and the sensor (e.g., to facilitate indoor positioning as well as data transfer).

[0032] Next, at operation 310, spatial setup is performed. In embodiments, spatial setup can include tracking of the user (e.g., by a mobile device) and a boundary defining an area where VR use is safe. This boundary can then be displayed to the user during VR use, allowing the user to move freely within the boundary without bumping into nearby objects. In some embodiments, spatial setup can include designating a stand-still or seated state where the user is not expected to move, thereby avoiding the need for a VR boundary. In these embodiments, spatial setup can include designating a stationary point and determining the height of the VR HMD and the ground plane (e.g., via IPS tracking) so that virtual gameplay can be properly displayed.

[0033] Next, a VR program is launched, as indicated by operation 315. Any suitable VR program may be launched, such as a video game, a mapping and navigation application, an artwork application, a word processing application, etc. Launching the VR program may cause the VR HMD to begin displaying data that is processed by the computer system.

[0034] Next, at operation 320, sensors begin tracking the VR HMD and mobile device. Tracking can include utilizing IPS technology to determine the location of the VR HMD and mobile device in real time, allowing for updates to the location of the HMD and mobile device within the virtual reality. This can also track the user's position relative to a virtual boundary to prevent the user from colliding with objects.

[0035] User input is then received from the user on the mobile device, as indicated by operation 325. The user input is received by a computer system and can be used to control VR gameplay while the user is immersed in VR. Examples of input that can be used to control VR use include touch-based interactions, motion-based interactions, bezel-based interactions, and in-air interactions.

[0036] In embodiments, the display on the mobile device can be mirrored within the virtual environment to assist the user in selecting control options. For example, if a touchscreen GUI is displayed on the mobile device but is not visible to the user (because the user is wearing a VR HMD), the touchscreen GUI can be mirrored within the virtual environment so that the user can see the available control options on the touchscreen GUI within the virtual reality. In some embodiments, content of interest displayed on the mobile device can be displayed only within the virtual reality, as it does not need to be viewed outside of the virtual reality.

[0037] The VR content displayed on the VR HMD is then controlled based on the user input, as shown in operation 330. Examples of controls that may be achieved based on the received input include performing an action in the virtual reality (e.g., interacting with an object in VR, such as picking up an item), aiming, moving, panning, zooming, etc.

[0038] The operations described above may be completed in any order and are not limited to those described. Furthermore, some, all, or none of the operations described above may be completed and still be within the scope of the present disclosure.

[0039] 4 is a diagram illustrating various controls 400 that can be used to control VR content displayed on a VR HMD, according to an embodiment of the present disclosure. The controls 400 include touch controls 410, motion controls 440, bezel-based controls 460, and in-air controls 470.

[0040] Touch controls 410 are generally defined as controls initiated based on touch input received on a touchscreen. Any suitable touchscreen may be implemented, including resistive, surface acoustic wave, and capacitive touchscreens. Various types of touch controls may be implemented, and the controls shown are not intended to limit other touch controls that may be implemented. Touch controls 410 include, among others, tap controls 415, touch dial controls 420, drag controls 425, or pinch / spread controls 430, or combinations thereof.

[0041] The tap control 415 is a control that is initiated in response to a tap received from a user on the touchscreen. The tap control 415 can be used to interact with objects in the virtual reality (e.g., a tap on the screen can cause a corresponding action, such as "picking up" an item). Additionally, the tap control 415 can be used to zoom in the virtual reality (e.g., a tap or double tap can cause a zoom action). Additionally, the tap control 415 can be used to change one or more views in the virtual reality. As an example, a view in VR can be changed based on the location on the touchscreen where the tap is registered. In some embodiments, the tap control 415 can be used to control VR movement. For example, the tap control 415 can be used to generate smooth motion or teleportation movements in the virtual reality.

[0042] The touch dial control 420 is a type of control that generates a digital dial on a device's touchscreen. The location of a user's touch along the digital dial results in a particular output. For example, a digital dial may include various options arranged around the circumference of the dial. As an example, a first control C1 may be located at 0 degrees on the dial, a second control C2 may be located at 90 degrees on the dial, a third control C3 may be located at 180 degrees on the dial, and a fourth control C4 may be located at 270 degrees on the dial. In this example, if a user's touch contacts the dial at the 90-degree position, the second control C2 may be selected. Thus, touch dial controls may be particularly useful when attempting to select from multiple options (e.g., multiple items in an inventory, controls on a control panel, etc.). In embodiments, it may be useful to be able to see the touch dial, and therefore the touch dial may be displayed in virtual reality to assist the user in controlling it.

[0043] The drag control 425 includes inputs that are accepted based on dragging actions performed on a touchscreen. In embodiments, the drag control 425 can be used to control a view within a virtual reality. For example, the drag control can be used to aim and pan. In embodiments, the drag control 425 can also be useful for VR movement. For example, the drag control 425 can be used for "pull" movement (e.g., a user moving within a virtual reality by pulling on the virtual reality world in response to a drag control). In embodiments, the drag control 425 can be used to scroll through a list of available options (e.g., on a virtual scroll bar). The drag control 425 can also be used to control / manipulate objects within VR. However, the drag control 425 can be utilized in any other suitable manner.

[0044] The pinch / spread control 430 includes inputs that are accepted based on two contact points on a touchscreen being touched and then moved together or apart. As shown in the pinch / spread control 430 of FIG. 4, the pinch control involves moving two contact points on the touchscreen closer together, and the spread control involves moving two contact points apart. The pinch / spread control can be used for zooming within a virtual reality (e.g., pinching to zoom in and spreading to zoom out). However, the pinch / spread control can also be used in any other suitable manner. For example, the pinch control can be used for forward locomotion within a virtual reality, and the spread control can be used for backward locomotion within a virtual reality.

[0045] Motion controls 440 generally refer to inputs generated in response to specific movements of the mobile device. These include tilt controls 445 and inertial controls 450, among other types of motion controls 440. Tilt controls 445 can be generated in response to changing the angle of the mobile device (e.g., tilting the device up, down, left, or right). In embodiments, device tilt can be detected by a tilt sensor such as an accelerometer or gyroscope. In embodiments, tilt controls 445 can be used for aiming in virtual reality (e.g., changing the viewing angle of a first-person or third-person perspective). For example, the view in VR can directly reflect the orientation of the mobile device. In embodiments, tilt controls can be used for steering (e.g., in virtual racing, flying, and boating games). In embodiments, tilt controls can be used to perform actions. For example, tilting the mobile device can cause a specific action (e.g., sprinting) to be performed in VR. However, tilt controls 445 can also be utilized in any other suitable manner.

[0046] Inertial control 450 can be generated in response to device movement. In embodiments, device movement can be detected using an accelerometer or a gyroscope, or both. Inertial control 450 can be used to change the viewing angle (e.g., aim and pan) based on the movement of the mobile device. Additionally, inertial control can be used to perform actions (e.g., certain movements result in certain actions, such as "jump"). In embodiments, inertial control can be used to control the movement of actions in the left and right hands of a VR user. That is, movement of the mobile device (e.g., held by the left and right hands) can result in corresponding movements in the virtual reality.

[0047] Bezel-based controls 460 include controls that generate input in response to interaction with areas of a mobile device that are not included on the touchscreen (e.g., the bezel portion). Common bezel-based controls include interaction with a "home" button (shown in bezel-based controls 460 in FIG. 4), interaction with a "lock button," interaction with a volume control, etc. Bezel-based controls 460 can include touch-based input controls that are not included on the touchscreen portion of a mobile device. For example, the "home" button on a smartphone typically includes a touch sensor and can utilize touch control. Various controls can be performed using bezel-based controls. For example, if the bezel portion of a smart device includes a "home" button with a touchpad, the touch control 410 described above can be completed using the "home" button. Bezel-based controls 460 can be used to change the viewing angle, perform actions, and exercise, among other control options.

[0048] In-air controls 470 include controls that generate input in response to specific gestures (e.g., hand or body gestures) made above the screen of a mobile device. In-air controls 470 utilize advanced depth cameras (e.g., the front-facing camera of a mobile device) and sensors to capture gestures made in the air (above the mobile device) and enable them to be translated into specific inputs within virtual reality. Here, "in-air" generally refers to touchless inputs.

[0049] Air controls 470 can be used to control various aspects of VR. In some embodiments, air controls 470 can be used for movement. For example, a user can point a finger up on the screen to complete forward movement. In some embodiments, air controls 470 can be used to perform actions (e.g., a specific gesture, "fist," can perform a specific action, "crouch"). In some embodiments, air controls 470 can be used to control the viewing angle (e.g., waving a hand can pan in the direction of the hand wave, pointing a finger up on a mobile device can aim in the direction of the finger, etc.).

[0050] In some embodiments, the in-air controls 470 can be used to construct three-dimensional (3D) objects. For example, a user can trace a particular object in the air above the mobile device (i.e., an in-air trace pattern) to generate a corresponding 3D object in virtual reality (e.g., the square shown in the in-air controls 470 of FIG. 4). In embodiments, the size of the traced object can be increased in virtual reality.

[0051] In embodiments, the depth and / or speed at which mid-air interactions are performed affect the output on the VR HMD. For example, fast hand gestures can result in a faster pan speed, and slow hand gestures can result in a slower pan speed. Similarly, gestures near the screen can result in a zoom in, and gestures far from the screen can result in a zoom out.

[0052] In an embodiment, various different controls 400 can be used simultaneously to control virtual reality content displayed on the VR HMD. Reference is now made to various gameplay examples, including the various controls described above. In a first example, such as a first-person perspective game, tilt control 445 can be used to control the viewing angle of the VR gameplay, drag control 425 can be used to move objects within the VR gameplay, and bezel-based control 460 can be used to control user movement. Furthermore, in this example, tap control 415 can be used to complete an action. For example, a "double tap" can cause a first action (e.g., crouch) in the virtual reality, and a "hold and press" tap can cause a second action (e.g., change an item). Furthermore, dial control 420 can assist in option selection (e.g., "open a door" or "walk away") when faced with multiple options.

[0053] In a second example, such as a 3D map viewer (e.g., GOOGLE EARTH®), the drag control 425 can be used to pan (e.g., camera movement along a horizontal plane), and the pinch / spread control 430 can be used to zoom in and out. However, in some embodiments, the air control 470 can be used for pan and zoom control. For example, panning can be controlled by dragging a finger in the air. The speed at which the user's finger moves across the screen can be detected by the front camera and converted into a corresponding pan speed. In embodiments, the zoom factor can be controlled by the distance the user's hand is measured in the air above the mobile device. For example, if the user's hand is close to the mobile device, a zoom-in setting can be performed, and if the user's hand is far from the mobile device, a zoom-out setting can be performed.

[0054] In a third example, such as a building game (e.g., a house-building or city-building simulator), aerial controls 470 may be used to build a 3D structure (e.g., a skyscraper or a house). For example, an aerial trace of an object may be converted into a larger version of the house or skyscraper. In these embodiments, touch controls 410 may be used to select options (e.g., building materials or styles), motion controls 440 may be used to control the viewing angle, and bezel controls 460 may be used to generate movement (e.g., to allow a user to explore the constructed building).

[0055] In some embodiments, different device controls 400 can be implemented in VR gameplay. As an example, in an FPS game, motion controls 440 of a first device (e.g., a smartwatch) can be used to control aiming, touch controls 410 of the first device can be used to control a first set of actions (e.g., "crouch"), air controls 470 of a second device (e.g., a smartphone) can be used to control a second set of actions (e.g., "change item" or "sprint"), and bezel controls 460 of the second device can be used to control movement.

[0056] Although reference has been made to specific control configurations, any suitable control configuration may be implemented consistent with the scope of the present disclosure. The controls described above are exemplary only and are not intended to limit the present disclosure.

[0057] Although this disclosure includes detailed descriptions of cloud computing, it should be understood that implementation of the teachings described herein is not limited to cloud computing environments. Rather, embodiments of the present disclosure may be practiced in conjunction with any other type of computing environment now known or later developed.

[0058] Cloud computing is a service delivery model for enabling convenient, on-demand network access to a shared pool of configurable computing resources (e.g., networks, network bandwidth, servers, processing, memory, storage, applications, virtual machines, and services) that can be rapidly provisioned and released with minimal management effort or interaction with a service provider. This cloud model includes at least five characteristics, at least three service models, and at least four deployment models.

[0059] The characteristics are as follows:

[0060] On-Demand Self-Service: Cloud consumers can unilaterally provision computing capacity, such as server time or network storage, automatically as needed, without the need for human interaction with the service provider.

[0061] Broad network access: Computing power is available over the network and can be accessed through standard mechanisms, facilitating use by heterogeneous thin or thick client platforms (e.g., cell phones, laptops, PDAs).

[0062] Resource Pooling: Computing resources from a provider are pooled and offered to multiple consumers using a multi-tenant model. Various physical and virtual resources are dynamically allocated and reallocated based on demand. Consumers generally have no control or knowledge of the exact location of the resources they are provided with, resulting in a sense of location independence. However, consumers may be able to determine location at a higher level of abstraction (e.g., country, state, data center).

[0063] Rapid Elasticity: Computing capacity can be provisioned quickly and elastically, sometimes automatically, to instantly scale out and quickly release to instantly scale in. To the consumer, the computing power available for provisioning often appears unlimited, and can be purchased at any time and in any quantity.

[0064] Metered Services: Cloud systems leverage measurement capabilities at a level of abstraction appropriate to the type of service (e.g., storage, processing, bandwidth, active user accounts) to automatically control and optimize resource usage. Resource usage can be monitored, controlled, and reported to provide transparency to both providers and consumers of utilized services.

[0065] The service model is as follows:

[0066] Software as a Service (SaaS): The functionality offered to the consumer is the availability of a provider's applications running on a cloud infrastructure that can be accessed from a variety of client devices through a thin client interface such as a web browser (e.g., webmail). The consumer does not manage or control the underlying cloud infrastructure, including the network, servers, operating systems, storage, or even individual application functionality, except for limited user-specific application configuration settings.

[0067] Platform as a Service (PaaS): The capability offered to consumers is to deploy applications they create or acquire using programming languages and tools supported by the provider onto a cloud infrastructure. The consumer does not manage or control the underlying cloud infrastructure, including the network, servers, operating systems, or storage, but does have control over the deployed applications and, in some cases, the configuration of their hosting environment.

[0068] Infrastructure as a Service (IaaS): The functionality offered to consumers is the provisioning of processors, storage, networking, and other basic computing resources on which they can deploy and run any software, including operating systems and applications. The consumer does not manage or control the underlying cloud infrastructure, but has control over the operating system, storage, and deployed applications, and in some cases partial control over some network components (e.g., host firewalls).

[0069] The deployment model is as follows:

[0070] Private Cloud: This cloud infrastructure is dedicated to a specific organization and can be managed by that organization or a third party, and can exist on-premise or off-premise.

[0071] Community Cloud: This cloud infrastructure is shared by multiple organizations to support a specific community with common concerns (e.g., mission, security requirements, policies, and compliance). This cloud infrastructure can be managed by those organizations or a third party and can exist on-premises or off-premises.

[0072] Public cloud: This cloud infrastructure is available to the general public or large industry organizations and is owned by an organization that sells cloud services.

[0073] Hybrid cloud: This cloud infrastructure combines two or more cloud models (private, community, or public), each of which retains its inherent nuances but is bound by standards or specific technologies that enable data and application portability (e.g., cloud bursting for load balancing between clouds).

[0074] A cloud computing environment is a service-oriented environment that emphasizes statelessness, low coupling, modularity, and semantic interoperability. At the core of cloud computing is an infrastructure that includes a network of interconnected nodes.

[0075] 5 illustrates an exemplary cloud computing environment 510. As shown, the cloud computing environment 510 includes one or more cloud computing nodes 500, with which local computing devices used by cloud consumers (e.g., PDA or cell phone 500A (e.g., controller 215), desktop computer 500B (e.g., computer system 110 or 235, or both), laptop computer 500C (e.g., computer system 110 or 235, or both), or automobile computer system 500N, or combinations thereof) can communicate. The nodes 500 can communicate with each other. The nodes 500 can be physically or virtually grouped (not shown) in one or more networks, such as, for example, a private, community, public, or hybrid cloud, or combinations thereof, as described above. This enables the cloud computing environment 510 to provide infrastructure, platform, or software as a service, or combinations thereof, for which cloud consumers do not need to maintain resources on their local computing devices. It should be understood that the types of computing devices 500A-N shown in FIG. 5 are merely exemplary, and that the computing nodes 500 and cloud computing environment 510 can communicate with any type of electronic device via any type of network or network-addressable connection (e.g., using a web browser), or both.

[0076] A set of functional abstraction layers provided by the cloud computing environment 510 (FIG. 5) is now shown in FIG. 6. It should be understood in advance that the components, layers, and functions shown in FIG. 6 are merely exemplary, and embodiments of the present invention are not limited thereto. As shown, the following layers and corresponding functions are provided:

[0077] Hardware and software layer 600 includes hardware and software components. Examples of hardware components include a mainframe 602, a reduced instruction set computer (RISC) architecture-based server 604, a server 606, a blade server 608, storage devices 610, and a network and network components 612. In some embodiments, software components include network application server software 614 and database software 616.

[0078] The virtualization layer 620 provides an abstraction layer from which virtual entities such as virtual servers 622, virtual storage 624, virtual networks including virtual private networks 626, virtual applications and operating systems 628, and virtual clients 630 can be provided.

[0079] By way of example, the management layer 640 may provide the following functionality: Resource provisioning 642 enables dynamic procurement of computing and other resources utilized to execute tasks within the cloud computing environment. Metering and pricing 644 enables cost tracking as resources are utilized within the cloud computing environment and billing or invoicing for the consumption of these resources. By way of example, these resources may include application software licenses. Security enables the identification and verification of cloud consumers and tasks, as well as protection for data and other resources. User portal 646 provides consumers and system administrators with access to the cloud computing environment. Service level management 648 enables the allocation and management of cloud computing resources so that requested service levels are met. Service level management 648 may allocate appropriate processing power and memory to process static sensor data. Service level agreement (SLA) planning and fulfillment 650 enables the advance arrangement and procurement of cloud computing resources anticipated for future needs according to SLAs.

[0080] The workload layer 660 provides examples of functionality that can be leveraged in a cloud computing environment. Examples of workloads and functionality that can be provided from this layer include mapping and navigation 662, software development and lifecycle management 664, indoor positioning 666, data analytics processing 668, transaction processing 670, and virtual reality management 672.

[0081] 7 is a schematic block diagram of an example computer system 701 (e.g., computer system 110, controller 215, and computer system 235) that can be used to implement (e.g., using one or more processor circuits of a computer or computer processor) one or more of the methods, tools, modules, and any associated functionality described herein, according to embodiments of the present disclosure. In some embodiments, the major components of computer system 701 can include one or more CPUs 702, a memory subsystem 704, a terminal interface 712, a storage interface 714, an I / O (input / output) device interface 716, and a network interface 718, all of which can be communicatively coupled, directly or indirectly, for communication between components via a memory bus 703, an I / O bus 708, and an I / O bus interface unit 710.

[0082] Computer system 701 may include one or more general-purpose programmable central processing units (CPUs) 702A, 702B, 702C, and 702D (collectively referred to herein as CPUs 702). In some embodiments, computer system 701 may include multiple processors, as is common in larger systems, although in other embodiments, computer system 701 may instead be a single-CPU system. Each CPU 702 may execute instructions stored in memory subsystem 704. Each CPU 702 may also include one or more levels of on-board cache.

[0083] The system memory 704 may include computer system-readable media such as volatile memory, such as random access memory (RAM) 722 and cache memory 724. The computer system 701 may further include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, the storage system 726 may be provided for reading from and writing to non-removable, non-volatile magnetic media, such as a “hard drive.” Although not shown, a magnetic disk drive for reading from and writing to a removable, non-volatile magnetic disk (e.g., a floppy disk) and an optical disk drive for reading from and writing to a removable, non-volatile optical disk (e.g., a CD-ROM, DVD-ROM, or other optical media) may be provided. Furthermore, the memory 704 may include flash memory (e.g., a flash memory stick drive or flash drive). Memory devices may be connected to the memory bus 703 by one or more data media interfaces. The memory 704 may include at least one program product having a set (e.g., at least one) of program modules configured to perform the functions of various embodiments.

[0084] One or more programs / utilities 728, each having at least one set of program modules 730, can be stored in memory 704. The programs / utilities 728 can include a hypervisor (also called a virtual machine monitor), one or more operating systems, one or more application programs, other program modules, and program data. Each of the operating system, one or more application programs, other program modules, and program data, or any combination thereof, can include a network environment implementation. The programs 728 and / or program modules 730 generally perform the functions or methods of various embodiments.

[0085] Although memory bus 703 is illustrated in FIG. 7 as a single bus structure providing a direct communication path between CPU 702, memory subsystem 704, and I / O bus interface 710, memory bus 703 may, in some embodiments, include multiple distinct buses or communication paths. These buses or communication paths may be arranged in any of a variety of configurations, such as point-to-point links in a hierarchical star or web configuration, multiple hierarchical buses, parallel and redundant paths, or any other suitable type of configuration. Furthermore, while I / O bus interface 710 and I / O bus 708 are each illustrated as single units, computer system 701 may, in some embodiments, include multiple I / O bus interface units 710, multiple I / O buses 708, or both. Furthermore, although multiple I / O interface units are illustrated, which separate I / O bus 708 from the various communication paths leading to the various I / O devices, in other embodiments, some or all of the I / O devices may be directly connected to one or more system I / O buses.

[0086] In some embodiments, computer system 701 may be a multi-user mainframe computer system, a single-user system, or a server computer or similar device that has little or no direct user interface but receives requests from other computer systems (clients). Further, in some embodiments, computer system 701 may be implemented as a desktop computer, a portable computer, a laptop or notebook computer, a tablet computer, a pocket computer, a telephone, a smartphone, a network switch or router, or any other suitable type of electronic device.

[0087] It should be noted that Figure 7 is intended to illustrate representative major components in an example computer system 701. In some embodiments, the individual components may be more complex or simpler than those shown in Figure 5. Also, components other than or in addition to those shown in Figure 5 may be present, and the number, type, and configuration of such components may vary.

[0088] As described in more detail herein, it is contemplated that some or all of the operations in some embodiments of the methods described herein may be performed in other order or not at all. Additionally, multiple operations may be performed simultaneously or as part of a larger process.

[0089] The present disclosure may be a system, a method, or a computer program product, or a combination thereof. The computer program product may include a computer-readable storage medium having stored thereon computer-readable program instructions for causing a processor to perform aspects of the present disclosure.

[0090] A computer-readable storage medium may be a tangible device capable of retaining and storing instructions for use by an instruction execution device. The computer-readable storage medium may be, by way of example, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or a suitable combination thereof. More specific examples of computer-readable storage media include portable computer diskettes, hard disks, RAM, ROM, EPROM (or flash memory), SRAM, CD-ROMs, DVDs, memory sticks, floppy disks, mechanically encoded devices having instructions recorded on punch cards or ridge-in-groove structures, or the like, and suitable combinations thereof. As used herein, a computer-readable storage medium should not be construed as a transitory signal per se, such as an electric wave or other freely propagating electromagnetic wave, an electromagnetic wave propagating through a waveguide or other transmission medium (e.g., light pulses passing through a fiber optic cable), or an electrical signal transmitted over a wire.

[0091] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to each computer / processing device. Alternatively, they can be downloaded to an external computer or external storage device via a network (e.g., the Internet, a LAN, a WAN, or a wireless network, or a combination thereof). The network can include copper transmission cables, optical fiber transmissions, wireless transmissions, routers, firewalls, switches, gateway computers, or edge servers, or a combination thereof. A network adapter card or network interface within each computer / processing device receives the computer-readable program instructions from the network and transfers the computer-readable program instructions to a computer-readable storage medium in the respective computer / processing device for storage.

[0092] The computer-readable program instructions for carrying out the operations of the present disclosure can be either assembler instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state-setting data, or source or object code written in any combination of one or more programming languages, including object-oriented programming languages such as Smalltalk and C++, and conventional procedural programming languages such as the "C" programming language and similar programming languages. The computer-readable program instructions can be executed entirely on the user's computer as a standalone software package, partially on the user's computer, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the latter case, the remote computer can be connected to the user's computer via any type of network, including a LAN or WAN, or can be connected to an external computer (e.g., via the Internet using an Internet Service Provider). In some embodiments, electronic circuitry, including, for example, programmable logic circuits, field programmable gate arrays (FPGAs), programmable logic arrays (PLAs), can execute computer-readable program instructions by utilizing state information of the computer-readable program instructions to customize the electronic circuitry for purposes of carrying out aspects of the present disclosure.

[0093] Aspects of the present disclosure are described herein with reference to flowchart and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the present disclosure. Each block of the flowchart and / or block diagram, and combinations of blocks in the flowchart and / or block diagram, can be implemented by computer-readable program instructions.

[0094] These computer-readable program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus to produce a machine, whereby the instructions, executed by the processor of such computer or other programmable data processing apparatus, create means for performing the functions / acts specified in one or more blocks of the flowcharts and / or block diagrams. These computer-readable program instructions can also be stored on a computer-readable storage medium that can instruct a computer, programmable data processing apparatus, or other device, or combination thereof, to function in a particular manner. The computer-readable storage medium having the instructions stored thereon thereby constitutes an article of manufacture including instructions for performing aspects of the functions / acts specified in one or more blocks of the flowcharts and / or block diagrams.

[0095] Additionally, computer-readable program instructions may be loaded into a computer, other programmable device, or other device and a series of operational steps executed on the computer, other programmable device, or other device to create a computer-implemented process, whereby the instructions executing on the computer, other programmable device, or other device perform the functions / operations identified in one or more blocks in the flowcharts and / or block diagrams.

[0096] The flowcharts and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of instructions, including one or more executable instructions for performing specific logical functions. In some other implementations, the functions shown in the blocks may be executed in an order different from that shown in the figures. For example, two blocks shown in succession may in fact be executed substantially concurrently, or the blocks may be executed in reverse order, depending on the functionality involved. Note that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented by a dedicated hardware-based system that performs specific functions or operations or executes a combination of dedicated hardware and computer instructions.

[0097] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the invention. As used herein, the singular forms "a," "an," and "the" are intended to include the plural unless the context clearly dictates otherwise. Furthermore, as used herein, the terms "include" and / or "including" specify the presence of stated features, integers, steps, operations, elements, or components, or combinations thereof, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, or groups or combinations thereof. In the above detailed description of exemplary embodiments of various embodiments, reference has been made to the accompanying drawings, in which like numerals represent like elements. The accompanying drawings, which form a part of this document, illustrate specific exemplary embodiments in which various embodiments may be practiced. These embodiments have been described in sufficient detail to enable those skilled in the art to practice the embodiments, but other embodiments may be utilized, and logical, mechanical, electrical, and other changes may be made without departing from the scope of the various embodiments. In the above description, numerous specific details are set forth to provide a thorough understanding of various embodiments. However, various embodiments may be practiced without these specific details. In other instances, well-known circuits, structures, and techniques have not been shown in detail to avoid obscuring the embodiments.

[0098] Various instances of the term "embodiment" as used herein do not necessarily refer to the same embodiment, but may. Any data and data structures illustrated or described herein are exemplary only; other embodiments may use different amounts of data, different types of data, different fields, different numbers and types of fields, different field names, different numbers and types of rows, records, entries, or different organizations of data. Furthermore, any data may be combined with logic, in which case separate data structures may not be required. Therefore, the above detailed description should not be construed in a limiting sense.

[0099] While various embodiments of the present disclosure have been described by way of example, they are not intended to be exhaustive or limited to these embodiments. It will be apparent to those skilled in the art that many modifications and variations are possible without departing from the scope and spirit of the described embodiments. The terms used herein have been selected to best explain the principles, practical applications, or technical improvements to commercially recognized technologies of the embodiments, or to enable those skilled in the art to understand the embodiments disclosed herein.

[0100] While the present disclosure has been described in connection with specific embodiments, it is anticipated that modifications and variations of these embodiments will be apparent to those skilled in the art. It is therefore intended that the appended claims be interpreted to cover all such modifications and variations that fall within the true scope of the present disclosure.

Claims

1. Establishing communication between a computer system, a virtual reality (VR) head-mounted display (HMD), and a mobile device; accepting user input at the mobile device configured to control VR content output by the computer system and displayed on a display of the VR HMD; Controlling the VR content displayed on the VR HMD based on the user input received at the mobile device; accepting a bezel-based control from a user of the mobile device; Controlling the user's movement within the VR content based on the bezel-based control; accepting touch controls from the user of the mobile device; In response to receiving the touch control, perform a first action within the VR content, the first action being different from the user's movement controlled based on the bezel-based control; and A method comprising:

2. the user input is an in-air control; The method of claim 1.

3. A specific gesture performed above the mobile device causes a specific action to be performed within the VR content. The method of claim 2.

4. accepting control in the air from a user of the mobile device; Controlling a viewing angle of the VR content based on the control in the air; and The method of claim 1 further comprising:

5. accepting control in the air from a user of the mobile device; performing a first action within the VR content based on the control in the air; accepting motion control from the user at a second mobile device; changing a viewing angle of the VR content based on the motion control; The method of claim 1 further comprising:

6. Virtual reality (VR) head-mounted display (HMD), Mobile devices and a computer system communicatively coupled to the VR HMD and the mobile device, the computer system including at least one memory component and at least one processor, the processor: accepting user input from the mobile device configured to control VR content displayed on a display of the VR HMD; controlling the VR content displayed on the VR HMD based on the user input received from the mobile device; configured to perform a method comprising: The method executed by the processor comprises: accepting touch controls from a user of the mobile device; Controlling the user's movement within the VR content based on the touch control; accepting motion control from a user of a second mobile device; Changing a viewing angle within the VR content based on the motion control; The system further includes:

7. the user input is an in-air control; The system of claim 6.

8. A specific gesture performed above the mobile device causes a specific action to be performed within the VR content. The system of claim 7.

9. The method executed by the processor comprises: capturing a control trace pattern in the air by a user of the mobile device using a front camera of the mobile device; constructing a three-dimensional virtual object in VR based on the control trace pattern in the air; capturing an in-air hand gesture by the user of the mobile device using the front camera of the mobile device; panning a camera view within the VR content based on the hand gesture in the air; The system of claim 6 further comprising:

10. The method executed by the processor comprises: accepting control in the air from a user of the mobile device; performing a first action within the VR content based on the control in the air; accepting bezel-based controls from the user at a second mobile device; changing a viewing angle of the VR content based on the bezel-based control; and The system of claim 6 further comprising:

11. The method executed by the processor comprises: capturing a first gesture by a user of the mobile device using a front camera of the mobile device; Controlling movement within the VR content based on the first gesture by the user; capturing a second gesture by the user of the mobile device using the front camera of the mobile device; changing a camera view within the VR content based on the second gesture; The system of claim 6 further comprising:

12. further comprising at least one sensor; The method executed by the processor comprises: tracking the mobile device with an indoor positioning system using the at least one sensor; updating a location of the mobile device within the VR content based on the tracking; and The system of claim 6 further comprising:

Images