Vestibular anchoring

Abstract

Methods and apparatus for presenting virtual reality images may include receiving head motion information from a position tracking system that tracks the position and orientation of a user's head, and receiving virtual motion information independent of the head motion information for use in a virtual reality simulation. The methods and apparatus may include generating a first scene of the virtual reality simulation based on the virtual motion information and the head motion information. The methods and apparatus may also include generating a set of visual cues based on the head motion information, and rendering a frame including the set of visual cues and the first scene of the virtual reality simulation. The methods and apparatus may include sending a frame of the virtual reality simulation for presentation on a display device.

Description

Technical Field

[0001] The present disclosure relates to computer graphics systems, and more particularly to presenting images on a display. Background Art

[0002] One area of ​​computing devices that has developed in recent years is gaming devices and virtual reality (VR) devices that use a graphics processing unit (GPU) to render graphics from a computing device to a display device based on rendering instructions received from the computing device. In gaming devices, a scene generated on a display device may be oriented or modified based on user input (e.g., movement of an external controller to cause movement of a scene orientation, introduction of items into a scene, etc.). Similarly, in VR devices, a scene generated on a display device may be oriented or modified based on user input, where the input may include detecting movement of a user's head (e.g., detected movement of a VR device (e.g., a head mounted display (HMD))).

[0003] A common problem in VR applications is the creation of conditions that may promote motion sickness, also known as simulation sickness. Individuals may be susceptible to motion sickness in a virtual reality environment because their external view of the world is removed and completely replaced by a simulated view that is affected by, but not completely controlled by, physical movement. The condition may result when the simulated view deviates from the view that the brain expects based on stimulation of other senses, most notably the vestibular system. As a result, when there is a discrepancy between the signals sent to the brain through the eyes and the inner ear, the user may have an uncomfortable experience when using a VR device. For example, when a user is viewing a VR scene that includes motion but the user is not physically moving, the difference in motion-related signals sent to the brain may cause the user to feel discomfort, nausea, fatigue, sweating, vertigo, or other motion sickness effects. Once initiated, motion sickness effects typically persist and even worsen until the inconsistency between the visual input and the signals from the inner ear is resolved.

[0004] Simulation sickness in virtual reality can be reduced or prevented by maximizing the accuracy of tracking head movement, maximizing the accuracy of predicting future movement, and minimizing the latency between the calculation of head pose and the display of the virtual scene corresponding to the pose. When simulated visual input closely matches physical input from the vestibular system, motion sickness may no longer be a factor for most users in virtual reality.

[0005] However, limiting the view displayed in a virtual reality environment to only the movement of the head may limit the freedom of movement traditionally provided by simulations that maintain independent manual input control (e.g., those found in video games with a first-person view). By enforcing strict adherence to head movement, the user may be restricted to the confines of the physical space containing the VR system; this in turn may limit the range of experiences that can be provided through virtual reality.

[0006] For example, while video game simulations often allow users to run for miles across alien worlds while sitting on a comfortable couch and pressing the thumbsticks of a game controller, the same experience may not be translatable into virtual reality when movement is limited to the head unless wearing a standalone VR rig with miles of open space that matches the simulated terrain.

[0007] Therefore, there is a need in the art for improvements in presenting VR images on displays. Summary of the invention

[0008] A simplified overview of one or more implementations of the present disclosure is presented below to provide a basic understanding of such implementations. This content is not an extensive overview of all contemplated implementations, and is neither intended to identify key or important elements of all implementations nor to describe the scope of any or all implementations. Its sole purpose is to present some concepts of one or more implementations of the present disclosure in a simplified form as a prelude to a more detailed description presented later.

[0009] The present disclosure relates to virtual reality simulations having a displayed view that reflects accurate head movements, combined with incorporating manual input controls for additional offset movements. Anchoring the displayed view to a scene that matches the movements sensed by the user's vestibular system, while also including a scene with virtual movements based on manual input, can provide a realistic virtual environment while reducing or preventing simulation sickness inherent in existing solutions due to the motion difference between actual user movements and virtual movements.

[0010] One example implementation involves a computer device. The computer device may include: a memory storing at least one application configured to run a virtual reality simulation; a processor in communication with the memory and configured to execute the at least one application to run the virtual reality simulation; an operating system in communication with the memory and the processor. The operating system, the processor, and the at least one application may be operable to: receive head motion information from a position tracking system that tracks a position and orientation of a user's head; receive virtual motion information independent of the head motion information for use in the virtual reality simulation; generate a first scene of the virtual reality simulation based on the virtual motion information and the head motion information; generate a set of visual cues based on the head motion information; render a frame including the set of visual cues and the first scene of the virtual reality simulation; and send the frame of the virtual reality simulation for presentation on a display device.

[0011] Another example implementation relates to a method for presenting a virtual reality image. The method may include receiving head motion information from a position tracking system that tracks the position and orientation of a user's head. The method may also include receiving virtual motion information independent of the head motion information for use in a virtual reality simulation. Additionally, the method may include generating a first scene of the virtual reality simulation based on the virtual motion information and the head motion information. The method may include generating a set of visual cues based on the head motion information, and rendering, at an operating system executed on a computer device, a frame of the first scene including the set of visual cues and the virtual reality simulation. The method may also include sending a frame of the virtual reality simulation for presentation on a display device.

[0012] Another example implementation relates to a computer-readable medium storing instructions executable by a computer device. The computer-readable medium may include at least one instruction for causing the computer device to receive head motion information from a position tracking system that tracks the position and orientation of a user's head. The computer-readable medium may include at least one instruction for causing the computer device to receive virtual motion information independent of the head motion information for a virtual reality simulation. The computer-readable medium may include at least one instruction for causing the computer device to generate a first scene of the virtual reality simulation based on the virtual motion information and the head motion information. The computer-readable medium may include at least one instruction for causing the computer device to generate a set of visual cues based on the head motion information. The computer-readable medium may include at least one instruction for causing the computer device to render a frame including the set of visual cues and the first scene of the virtual reality simulation. The computer-readable medium may include at least one instruction for causing the computer device to send a frame of the virtual reality simulation for presentation on a display device.

[0013] Additional advantages and novel features related to the implementation of the present disclosure will be set forth in part in the following description, and in part will become more apparent to those skilled in the art upon examination of the following or learning through practice thereof. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] In the attached picture:

[0015] Figure 1 is a schematic block diagram of an example device according to an implementation of the present disclosure;

[0016] Figure 2 is a schematic diagram of an optical device and a display panel of a head-mounted display for displaying virtual reality images according to an implementation of the present disclosure;

[0017] Figure 3 is a schematic diagram of a virtual reality simulation with two scenes according to an implementation of the present disclosure;

[0018] Figure 4 is a flow chart of a method for displaying a virtual reality image according to an implementation of the present disclosure;

[0019] Figure 5 is a flow chart of a method for displaying a virtual reality simulation having two scenes according to an implementation of the present disclosure; and

[0020] Figure 6 is a schematic block diagram of an example device according to an implementation of the present disclosure. DETAILED DESCRIPTION

[0021] The present disclosure relates to devices and methods for reducing or preventing simulation sickness when viewing a VR scene using a VR device. As used herein, a VR device may include a device that generates and / or displays a virtual reality image (e.g., from at least one virtual environment input), a mixed reality (MR) image (e.g., from at least two virtual environment inputs), and / or an augmented reality (AR) image (e.g., from at least one virtual environment input and one real environment input).

[0022] Vestibular anchoring is a technique in which a set of visual cues is provided to a user in a virtual reality environment to allow the application of virtual motion effects that can be obtained from sources other than the tracking system in a head-mounted display (HMD). Visual cues and relative motion of visual cues can be obtained based on head motion information received from a position tracking system (e.g., the position and / or orientation of the user's head). For example, the position tracking system can be configured to track the movement of an HMD device, and can be independent of the HMD device or mounted on the HMD device, and can therefore be referred to as an HMD-based motion source. Additionally, the visual cues can generate visual signals to the brain that can be consistent with signals generated by the vestibular system in response to the user's physical movements. Based on HMD tracking information (e.g., motion that substantially matches vestibular sensory input), visual cues using HMD-based motion can then supplement the virtual reality environment presented to the user, which itself includes a combination of virtual motion based on virtual motion information received from one or more non-HMD sources (e.g., manual input from a user pressing a button or moving a switch on a game controller, or from an application generating a virtual reality environment) and received from an HMD-based motion source. Visual cues corresponding to head motion information can be presented anywhere in the virtual reality environment so that the fusion of complementary visual and vestibular sensory inputs in the brain can overcome the conflicting display of virtual motion that might otherwise manifest as motion sickness.

[0023] Thus, the present disclosure is directed to providing apparatus and methods for providing visual cues (e.g., vestibular anchoring cues) that correspond to signals from the vestibular system, and display of a primary view of a different, conflicting virtual reality environment that corresponds to virtual motion. Such cues can take any form and exist anywhere within the visible range of the user (e.g., a grid intermingled throughout the scene). In some cases, cues can be placed in areas of peripheral vision where the user's perception of detail is less acute. For example, a virtual reality environment (whose virtual user motion includes non-HMD-based effects) can be displayed within a saucer-shaped area at the center of vision of each eye of the user, and a set of visual cues whose motion is based solely on head motion information (e.g., HMD-based motion) can be confined to an area of ​​the peripheral ring surrounding the butterfly. Anchoring cues in the peripheral vision area can be less distracting while still providing the necessary signals to the brain.

[0024] In addition, the apparatus and methods of the present disclosure may include: dynamically introducing and removing vestibular anchoring cues based on motion effects originating from sources other than the user's head tracking (adding cues when such effects are present, and removing them when head tracking alone is responsible for the motion); incorporating anchoring cues into the formation of a virtual reality scene (involving possible transition effects); the possibility of anchoring cues originating from one or more live camera feeds and adjusting the cue strength based on the comfort needs of the individual user.

[0025] In one implementation, the displayed virtual environment can be a VR simulation with two independent scenes that are fused together into a displayed scene presented to the user. For example, a first of the two independent scenes can correspond to the main VR scene and have a first (virtual) amount of motion, and a second scene can correspond to the above-mentioned visual cues and have a second amount of motion, the second amount of motion being controlled by head motion information received from a tracking system (e.g., head motion corresponding to the user's real-world motion or HMD tracking information). The second scene can correspond to a different virtual scene, or to an independent view of the original virtual environment. Thus, the second scene can include visual cues to match the user's vestibular sensory input (based solely on head motion information), which can cause the displayed scene to suppress or reduce motion sickness.

[0026] In other words, a first scene of a VR simulation may have a first view area with a first amount of motion that tracks movement of the virtual scene (where such motion is some fusion of a motion source from the simulation itself, head motion or HMD tracking, and / or user input control). At the same time, a second scene of the VR simulation may have a second view area different from the first view area, and the second scene may have a second amount of motion that is independent of the first amount of motion of the first view area, and instead it only tracks the movement of the HMD (HMD tracking information or head motion information from a tracking system), the movement of the HMD being the movement sensed by the user's inner ear. As described above, in some implementations, the second scene may be presented in a peripheral area surrounding the first scene. Thus, the described apparatus and methods allow a VR simulation to include one area with independent motion in the virtual world and another area with motion driven by real-world motion. By providing a coherent sensory input to the user, motion sickness that would otherwise be produced by conflicting views of a virtual reality scene may be reduced and / or prevented.

[0027] Reference now Figure 1, an example system 100 for use in conjunction with presenting a VR image may include a computer device 102 in communication with one or more display devices 106, the VR image including one area with independent motion in a virtual world and another area with motion driven only by real-world motion. The computer device 102 may execute at least one application 10 to run a virtual reality simulation 22, which generates and transmits image data (e.g., display frames 25) for displaying one or more VR images, defining one or more scenes of the virtual reality simulation 22 on the display device 106. The display device 106 may include, for example, a head-mounted display (HMD) device, and may have one or more display panels 32 (e.g., light emitting diodes (LEDs), organic LEDs (OLEDs), liquid crystal displays (LCDs), etc.) capable of outputting (multiple) display frames 25 for viewing by a user of the system 100.

[0028] The computer device 102 may communicate with the position tracking system 107 via the network 104 and / or through a wired or wireless connection to obtain head movement information 14 for application to the virtual reality simulation 22. The position tracking system 107 may include, but is not limited to, one or more sensors (e.g., cameras), emitters (e.g., IR LEDs), inertial devices (accelerometers and / or gyroscopes), and / or any other position sensing system capable of detecting the orientation, position, and / or movement of the user's head of the system 100 and / or display device 106 (e.g., HMD device), where the user is looking (e.g., the gaze direction of the eyes), and / or the user's focus. For example, in some implementations (e.g., "inside-out" implementations), the position tracking system 107 may include, but is not limited to, one or more depth cameras and one or more inertial measurement units (IMUs), tracking software, simultaneous localization and mapping algorithms, and one or more processors. In other implementations (such as "outside-in" implementations), the position tracking system 107 may include, but is not limited to, one or more sensors (e.g., cameras) for detecting one or more emitters (e.g., IR LEDs) located in the environment and one or more processors. Although the tracking system 107 is illustrated as being separate from the display device 106, it should be understood that the tracking system 107 can be remote from the display device 106 or located on the display device 106, or partially remote from the display device 106 and partially located on the display device 106.

[0029] Additionally, the computer device 102 may communicate with one or more external controllers 108 via the network 104 and / or through a wired or wireless connection to obtain virtual motion information 12 for application to the virtual reality simulation 22. The external controller 108 may include, but is not limited to, a game controller, a joystick, a keyboard, a mouse, or other input devices that can provide virtual motion information 12 to a virtual environment (e.g., a game or a virtual world) of the virtual reality simulation 22.

[0030] Optionally, in some cases, the computer device 102 may communicate with one or more cameras 109 via the network 104 and / or through a wired or wireless connection to obtain video stream information 111 for application to the virtual reality simulation 22. For example, in some implementations, the video stream information 111 from the camera(s) 109 may be video images of the real world that may be supplemented by virtual reality images from the virtual reality simulation 22 to define augmented reality images for presentation on the display device 106. Furthermore, in other implementations, for example, the video stream information 111 may be video images of the real world that may be used as input to create a computerized environment reconstruction, which may be used by the virtual reality simulation 22 for presentation on the display device 106 along with other virtual images.

[0031] The computer device 102 may include an operating system 110 executed by the processor 36 and / or the memory 38. The memory 38 of the computer device 102 may be configured to store data and / or computer executable instructions defining and / or associated with the operating system 110, and the processor 36 may execute such data and / or instructions to instantiate the operating system 110. Examples of the memory 38 may include, but are not limited to, computer usable memory types, such as random access memory (RAM), read-only memory (ROM), tape, disk, optical disk, volatile memory, non-volatile memory, and any combination thereof. Examples of the processor 36 may include, but are not limited to, any processor specifically programmed as described herein, including a controller, a microcontroller, an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), a system on a chip (SoC), or other programmable logic or state machine.

[0032] Computer device 102 may include any mobile or fixed computer device that can connect to a network. Computer device 102 may be, for example, a computer device such as a desktop or laptop computer or tablet computer, a cellular phone, a gaming device, a mixed reality or virtual reality device, a music device, a television, a navigation system, a camera, a personal digital assistant (PDA) or handheld device, or any other computer device having wired and / or wireless connection capabilities to one or more other devices and / or communication networks.

[0033] The operating system 110 may include a compositor 18 and a graphics processing unit (GPU) 24 operable to render one or more image frames and determine when to send the rendered image frames for presentation on the display device 106. In particular, the compositor 18 may receive one or more VR images and work with the GPU 24 to combine regions with different motions and generate one or more display frames 25, the VR image including one region with independent motion in the virtual world and another region with motion driven only by real-world motion. The GPU 24 and / or the compositor 18 may communicate with the display controller 26 to send the rendered image frames (e.g., display frames 25) for presentation on the display device 106.

[0034] Additionally, the operating system 110 may also include a display interface 28, which may be communicatively coupled to the processor 36 and / or the memory 38 for communicating with the display device 106 via a display port 30. The display port 30 may include various types of ports, including, but not limited to, a High Definition Multimedia Interface (HDMI) port, a Display Serial Interface (DSI) port, a Mobile Industry Processor Interface (MIPI) DSI port, a Universal Serial Bus (USB) port, a Firewire port, or other embedded display ports or external wired or wireless display ports that may allow, for example, communication via the network 104 between the computer device 102 and the display device 106.

[0035] The computer device 102 may also include one or more applications 10 that may be executed by the processor 36 to present one or more virtual reality images (e.g., display frames 25) on the display device 106. The one or more applications 10 executed by the processor 36 may also operate in conjunction with the operating system 110, the GPU 24, and / or the compositor 18 to run the virtual reality simulation 22 and generate (multiple) display frames 25 for presentation on the display device 106.

[0036] One or more applications 10 may include a virtual reality simulation 22 operable to receive motion input, such as head motion information 14 and / or virtual motion information 12, and render one or more image frames based on the received motion input for presentation on a display device 106. For example, one or more applications 10 may be a virtual reality application executable to run the virtual reality simulation 22 to generate a virtual world 11. The virtual world 11 may include, but is not limited to, a virtual reality game, a mixed reality game, an augmented reality game, an augmented reality environment (such as, but not limited to, a collaborative work environment), a virtual theater, or any other type of at least partially virtual environment. The one or more applications 10 may receive motion input from a position tracking system 107 and / or an external controller(s) 108. For example, the one or more applications 10 may receive virtual motion information 12 corresponding to movement in the virtual world 11 (e.g., of characters and / or objects in the virtual world 11) from the game controller(s) 108. The one or more applications 10 may also receive the virtual motion information 12 based on, for example, logic in the virtual reality simulation 22. Additionally, one or more applications 10 may receive head motion information 14 from the position tracking system 107 corresponding to movement of a display device 106 in the real world, for example in the form of an HMD worn by a user of the system 100 .

[0037] The one or more applications 10 may transform the virtual motion information 12 and / or the head motion information 14 into virtual coordinates. The one or more applications 10 may also combine the head motion information 14 and the virtual motion information 12 to determine a virtual pose (position and orientation) of a character and / or an object in the virtual world 11. In one implementation, the one or more applications 10 may render a first scene 16 of the virtual world 11 based on the virtual pose of the character and / or the object for the scene in the virtual world 11. The first scene 16 may correspond to an original view area of ​​the virtual scene and may track the motion of the virtual scene. The first scene 16 may be a main virtual scene.

[0038] The one or more applications 10 may generate the first scene 16, for example, based on a transformation matrix used in conventional scene rendering concatenated by a multiplication of W*V*P, where W is a world matrix, V is a view matrix, and P is a projection matrix. In one implementation, the one or more applications 10 may use the following transformation matrix when rendering the first scene 16:

[0039] W simulation *V head *P

[0040] Where W is the world matrix, P is the projection matrix, V simulation is the camera matrix formed by controller input and / or analog design, and V headis the tracked position of the user's head of system 100 and may incorporate an additional basis transform from the absolute tracking origin to the game space relative origin that may be established during setup. Additionally, V head Or P may apply an eye offset based on, for example, pupil distance. Thus, based on the above transformation matrix, one or more applications 10 may generate a first scene 16 and then provide the first scene 16 to the compositor 18.

[0041] The compositor 18 may also receive one or more visual cues 17 or a second scene 20 defining such visual cues 17, and may work with the GPU 24 to generate corresponding rendered frame(s) (e.g., a rendered first scene 113 and (multiple) rendered visual cues or second scene 114, which are combined and output as (multiple) display frames 25).

[0042] The one or more visual cues 17 or the second scene 20 may be generated by a first application in the one or more applications 10, a second application in the one or more applications 10, or by the operating system 110 based solely on the head motion information 14, such that the one or more visual cues 17 or the second scene 20 have an amount of motion that substantially matches real-world motion sensed by the vestibular system of a user of the system 100. The elements included in the one or more visual cues 17 or the second scene 20 may have a close relationship to the content from the first scene 16, or the elements may consist of completely separate collections of media.

[0043] Furthermore, in some cases, the one or more visual cues 17 or the second scene 20 defining such visual cues 17 may be generated by a different virtual reality simulation than the virtual reality simulation that generated the first scene 16 or an independent view of the same virtual reality simulation. In other words, the virtual reality simulation 22 may include a main virtual reality simulation 115 that generates the first scene 16 and an auxiliary virtual reality simulation 117 that generates the one or more visual cues 17 or the second scene 20. In this case, the auxiliary virtual reality simulation 117 may be a different virtual reality simulation than the main virtual reality simulation 115 or an independent view of the same virtual reality simulation.

[0044] Thus, in some cases, a first application of one or more applications 10 may generate a first scene 16 and one or more visual cues 17 or a second scene 20. For example, in this case, the first application of the one or more applications 10 may run a primary virtual reality simulation 115 that generates the first scene 16 and an auxiliary virtual reality simulation 117 that generates the one or more visual cues 17 or the second scene 20. In some implementations, the first application of the one or more applications 10 may then combine the first scene 16 and the one or more visual cues 17 or the second scene 20 and provide the combination to the compositor 18. In other implementations, the first application of the one or more applications 10 may send the first scene 16 and the one or more visual cues 17 or the second scene 20 to the compositor 18 for combination (with potential reprojection).

[0045] Furthermore, in some cases, a second application among the one or more applications 10 may generate the one or more visual cues 17 or the second scene 20, while a first application among the one or more applications 10 may generate the first scene 16. For example, in this case, the second application among the one or more applications 10 runs the auxiliary virtual reality simulation 117 to generate the one or more visual cues 17 or the second scene 20, and the first application among the one or more applications 10 runs the main virtual reality simulation 115 to generate the first scene 16. Then, the first application and the second application among the one or more applications 10 may separately provide the first scene 16 and the one or more visual cues 17 or the second scene 20 to the synthesizer 18, respectively.

[0046] Additionally, in some cases, the operating system 110 may generate the one or more visual cues 17 or the second scene 20, while the first application in the one or more applications 10 may generate the first scene 16. For example, in this case, the operating system 110 runs the auxiliary virtual reality simulation 117 to generate the one or more visual cues 17 or the second scene 20, and the first application in the one or more applications 10 runs the main virtual reality simulation 115 to generate the first scene 16. Then, the first application in the one or more applications 10 and the operating system 110 may separately provide the first scene 16 and the one or more visual cues 17 or the second scene 20 to the synthesizer 18, respectively.

[0047] As explained above, although the one or more visual cues 17 or the second scene 20 may be generated by the first application in the one or more applications 10, the second application in the one or more applications 10, or through the operating system 110, in order to simplify the description, the following examples will be discussed based on the implementation of the first application in the one or more applications 10. In other words, in the following description, the same application 10 generates the first scene 16 and the one or more visual cues 17 or the second scene 20.

[0048] A first application of the one or more applications 10 may receive head movement information 14 from the position tracking system 107, the head movement information 14 corresponding to movements of a user wearing a display device 106 in the form of an HMD in the real world.

[0049] The first application in the one or more applications 10 can transform the head movement information 14 into virtual coordinates to express the head position and orientation in the coordinate system of the virtual simulation. The transformation of the head position / orientation from the received original physical coordinates to the virtual coordinates can convert the head movement information 14 from the position tracking system 107 to a possibly modified "game space" pose established during the VR device setup (e.g., modifying the origin / center of the space from the origin / center specified by the absolute sensor positioning using the outside-in tracking system to the origin / center specified when the user selects the center of the room based on furniture or other objects in the room), and then possibly to a virtual user pose in the simulation (e.g., the tracking system can maintain coordinates in meters, while the game can construct its own coordinate system in feet, and the tracking system can also treat the forward direction Z as a positive value, while the simulation can use negative Z values ​​to express the forward direction). The first application in the one or more applications 10 can generate (multiple) visual cues 17 or a second scene 20 based on the head movement information 14. The visual cue(s) 17 or the second scene 20 and the relative movement of the visual cue(s) 17 or the second scene 20 may be generated based on the head movement information 14 received from the display device 106 (e.g., HMD). The visual cue(s) 17 or the second scene 20 may generate a visual signal to the brain that may be consistent with the signal generated by the vestibular system in response to the user's physical movement.

[0050] In one implementation, visual cue(s) 17 may be presented anywhere in the first scene 16 such that fusion of complementary visual input and vestibular sensory input in the brain can overcome conflicting displays of virtual motion (relative to real-world motion represented by head motion information 14) that might otherwise manifest as motion sickness.

[0051] In one implementation, a first application in one or more applications 10 may render visual cues 17 in the form of a second scene 20 such that the second scene 20 may include motion that matches the user's vestibular sensory input. In this way, the second scene 20 may provide the visual cues 17 necessary to suppress and / or prevent the user's motion sickness. In this implementation, the second scene 20 may correspond to a virtual scene different from the virtual scene of the first scene 16, where the elements may be composed of completely separate media collections. Although the second scene 20 may include any type of media, content, or composition, examples of the second scene 20 composition may include, but are not limited to: a virtual lobby area, a virtual theater environment, or even a display of the user's current physical environment. The user's current physical environment may be implemented, for example, by spatial reconstruction of a reprojected feed from an external camera (or cameras) 109 that is in communication with the computer device 102 and / or present on the display device 106, or a transparent view of the real world for the display device 106 with AR functionality. The second scene 20 may be constructed in such a way that a strong stereoscopic visual cue with detailed geometric and texture structures is provided to the user. For example, the second scene 20 may include geometric details that provide depth relative to the walls of the virtual environment and / or visual cues that indicate movement in the virtual environment. Thus, the second scene 20 may include motion independent of the first scene 16, and the second scene 20 may include motion tracking motion sensed by the user's inner ear, for example, based on head motion information 14 from the display device 106 (e.g., HMD).

[0052] In one implementation, a first application in the one or more applications 10 may use the following transformation matrix when rendering for each eye for the second scene 20:

[0053] W*V head *P

[0054] Where W is the world matrix, P is the projection matrix, V head is the tracked position and orientation of the head, and may incorporate an additional basis transform from the absolute tracking origin to the game space relative origin that may be established during setup. Additionally, V head Or P can apply eye offset based on, for example, pupil distance. Therefore, based on the above transformation matrix, a first application in one or more applications 10 (or a second application in one or more applications 10 or operating system 110) can generate one or more visual cues 17 or a second scene 20 with movement based only on head movement information 14 to substantially match the movement sensed by the vestibular system of the user of system 100.

[0055] In some implementations, as described above, the compositor 18 may include transition effects 121 between the first scene 16 and the second scene 20. The transition effects 121 may make the discontinuity between elements in the first scene 16 and the second scene 20 less visually noticeable, and may also reduce annoying “window violation” stereo artifacts between the first scene 16 and the second scene 20. The transition effects 121 may include, but are not limited to: cross-fading with dimming in the transition area (e.g., the area between the first scene 16 and the second scene 20), introduction of a blurred boundary ring, radial gradient dimming of the entire surrounding scene (e.g., brightest at the edges and gradually darker toward the center), dark boundaries (e.g., black or other dark colors between areas), colored rings (e.g., glowing rings between areas), and blending images from the first scene 16 into the second scene 20.

[0056] A first application of the one or more applications 10 may transmit instructions and data for rendering a scene from a virtual reality simulation 22 to a GPU 24. The GPU 24 may generate a rendered virtual first scene 113 (based on the virtual first scene 16) and (multiple) rendered virtual visual cues or second scenes 114 (based on (multiple) virtual cues 17 or virtual second scenes 20) and / or it may render a unified composition of the first scene 16 and (multiple) visual cues 17 or second scenes 20 to generate (multiple) display frames 25. The compositor 18 may optionally combine the rendered first scene 113 with the rendered second scene 114, and may optionally add transition effects 121 to such rendering, thereby generating (multiple) display frames 25. The compositor 18 and / or the GPU 24 may determine when to send the (multiple) display frames 25 containing the combined scene to the display controller 26 for forwarding to and presentation on the display device 106. The rendering operations performed by the GPU 24 may be computationally intensive. The time to complete the rendering operation may depend on the quality of the display image being generated. For example, the rendering time may be based on factors such as resolution and color format, as well as the quality of various visual effects (e.g., shadows or particles). The quality of the image may be adjusted based on hardware capabilities so that the GPU 24 can generate new display frames at the refresh rate of the display device 106. For example, the refresh rate of a virtual reality (e.g., a virtual reality simulation 22) may be 90 Hz, resulting in a refresh every 11 milliseconds. The synthesizer 18 and / or the GPU 24 may communicate with the display controller 26 to send the rendered display frame 25 for presentation on the display device 106.

[0057] The display controller 26 may inform the display interface 28 to transmit the rendered display frame 25 to the display device 106 for presentation on the display device 106 .

[0058] Thus, the display frame 25 presented on the display panel 32 may include a rendered first scene 113 (corresponding to the virtual first scene 16), the first scene 113 having a motion based on the virtual motion information 12 and the head motion information 14. The rendered first scene 113 may optionally include one or more rendered visual cues 114 (based on the virtual visual cues 17) having a motion based only on the head motion information 14. Alternatively, the presented display frame 25 may include the rendered first scene 113 and the rendered second scene 114 (based on the virtual second scene 20), the rendered second scene 114 including or defining one or more visual cues 17. In either case, the rendered first scene 113 may correspond to a central area or a focal area viewed by the user in a view area on the display panel 32 (and may have, for example, a certain amount of virtual motion based on the virtual motion information 12), and the rendered visual cues or second scene 114 may be an area outside the center or focal area in the view area (and having, for example, independent motion associated with the head motion information 14).

[0059] The relative diameters of the display areas bounding the rendered first scene 113 and the (multiple) rendered visual cues or the second scene 114 may depend on the display area of ​​the display panel 32. Additionally, the relative diameters of the bounded areas of the rendered first scene 113 and the (multiple) rendered visual cues or the second scene 114 may be controlled by the user based on the user's susceptibility to motion sickness. For example, the user may expand or minimize the rendered first scene 113 based on susceptibility to motion sickness. If the user has a lower susceptibility to motion sickness, the user may choose to expand the rendered first scene 113 to have a larger viewing area and minimize the viewing area of ​​the (multiple) rendered visual cues or the second scene 114. However, if the user has a higher susceptibility to motion sickness, the user may choose to increase the viewing area of ​​the (multiple) rendered visual cues or the second scene 114 while reducing the viewing area of ​​the rendered first scene 113.

[0060] In other words, the presented display frame 25 may include a rendered first scene 113 having a first display area, the first display area having a motion amount that tracks the movement of the virtual scene. At the same time, the rendered visual clues or second scene 114 may be around the rendered first scene 113 and may have a second display area that is different from the first display area. The peripheral area of ​​the rendered visual clues or second scene 114 may have a motion amount that depends only on the movement of the viewer and may therefore be independent of the movement of the first virtual scene. Specifically, the peripheral area may have a motion amount that tracks the movement sensed by the user's inner ear, for example, based on the motion input from the HMD device (e.g., head motion information 14 from the position tracking system 107). In this way, the presented display frame (s) 25 may improve user comfort by reducing conflicting signals from the user's eyes and inner ears, thereby reducing or eliminating nausea, motion sickness, or vertigo.

[0061] An example use case may include a user playing a virtual game and the user uses an external controller 108 to move a character in the virtual game (e.g., virtual reality simulation 22) forward while physically remaining still (e.g., without head movement information 14 from position tracking system 107). Display frame 25 may include a rendered first scene 113 in which the character in the virtual game moves forward, and a rendered second scene 114 including still visual cues 17. Still visual cues 17 may generate visual signals to the user's brain that may be consistent with signals generated by the vestibular system in response to the user remaining still, such that the fusion of complementary visual and vestibular sensory inputs in the brain may overcome the conflicting display of virtual motion that may otherwise manifest as motion sickness.

[0062] Another example use case may include: a user plays a virtual game and when the user's head rotates left and right (e.g., the head movement information 14 received from the position tracking system 107 is left and right), the user uses the external controller 108 to move the character in the virtual game forward. The display frame 25 may include a rendered first scene 113, in which the character in the virtual game moves forward in the virtual game while looking left and right. The display frame 25 may also include a rendered second scene 114, and the rendered second scene 114 includes a set of visual cues 17. Because the user is not physically moving forward, the visual cues 17 included in the rendered second scene 114 may not indicate forward motion, but may match the received left and right movement of the user's head rotation.

[0063] Yet another example use case may include a user playing a virtual game and the user moves a character in the virtual game forward by physically walking forward, and may use an external controller 108 to make the character in the virtual game look left or right. The display frame 25 may include a rendered first scene 113, in which the character moves forward based on head motion information 14 received from the position tracking system 107, and looks left and right in the virtual game based on virtual motion information 12 received from the external controller 108. The display frame 25 may also include visual cues 17 throughout the rendered second scene 114, the visual cues 17 reflecting the forward motion, tracking the user's physical movements.

[0064] When the application(s) 10 and / or the operating system 110 stop receiving virtual motion information 12 from the external controller 108 and / or the virtual reality simulation 22 after a predetermined period of time (e.g., the user virtually stops moving for 5 seconds except for head movement), the rendered first scene 113 can be dynamically expanded back to cover the entire display area of ​​the display panel 32. For example, the application(s) 10 or the operating system 110 can remove the visual cues 17 and / or the rendered second scene 114 from the presented display frame 25. The animated expansion of the rendered first scene 113 can occur after the virtual motion information 12 stops. The rendered first scene 113 can maintain complete coverage of the entire display area until another auxiliary independent motion influence (e.g., virtual motion information 12 from the external controller 108) is detected, and the rendered second scene 114 can become visible to the user again. The rendered second scene 114 can provide an immediate anchoring cue to the user through one or more visual cues 17.

[0065] Reference now Figure 2 , a conceptual diagram illustrating the operation of a lens and display of an example display device 106 (e.g., an HMD) according to an implementation. The display device 106 may include an optical device 204 (e.g., one or more lenses) for focusing the user's vision on one or more portions of the view area 27 of one or more display panels 32. For example, the display panel 32 may include a liquid crystal display (LCD) (which may include a light emitting diode (LED), a backlit LCD display), an organic LED (OLED) display, a digital light processing (DLP) display, etc. The display panel 32 may be based on the display controller 26 ( Figure 1 ) to display one or more images (e.g., left eye image and right eye image). In one implementation, the display controller 26 can provide multiplexed left eye image and right eye image to the display device 106 in a side-by-side arrangement. It should be understood that the left eye image and the right eye image can also be vertically multiplexed or time multiplexed. The display panel 32 can receive the display port 34 ( Figure 1) scans an image (eg, a rendered frame). As used herein, scanning may refer to the process of updating pixels on the display panel(s) 32.

[0066] The optics 204 may include left eye optics 206 for focusing the user's left eye on the left eye image, and right eye optics 208 for focusing the user's right eye on the right eye image. For example, the optics 206 may focus the user's eye on a central portion of each of the left eye image and the right eye image. The user's brain may combine the images viewed by each eye to create the perception that the user is viewing a 3D environment. For example, the left eye image and the right eye image may both include rendered frame content that may be perceived as a three-dimensional object.

[0067] The presented display frame 25 may include a rendered first scene 113 having a first view area 210 having an amount of motion that tracks the motion of the virtual scene of the virtual world 111 based at least on the virtual motion information 12. At the same time, the presented display frame 25 may include a rendered second scene 114 that is peripheral to the rendered first scene 113. The rendered second scene 114 may have a second view area 202 that is different in size (height and / or width) from the first view area 210. The peripheral area of ​​the view area 27 occupied by the rendered second scene 114 may have an amount of motion that is independent of the motion of the virtual scene (e.g., independent of the rendered first scene 113). In particular, the rendered second scene 114 may have an amount of motion based on the head motion information 14 from the position tracking system 107, thereby substantially matching the real-world motion sensed by the user's vestibular system.

[0068] refer to Figure 3 , illustrating that on the display device 106 ( Figure 1 ) is an example display frame 25 displayed on the display. The presented display frame 25 may include a rendered first scene 113, the rendered first scene 113 corresponding to the application 10 ( Figure 1 )'s virtual world 11( Figure 1 ) virtual scene. Additionally, the presented display frame 25 may include a rendered second scene 114 ( Figure 1 ), the rendered second scene 114 is based on the position tracking system 107 ( Figure 1 ) received head movement information 14( Figure 1) to track the user's movement. The presented display frame 25 may also include a transition effect 121 between the rendered first scene 113 and the rendered second scene 114, which makes the discontinuity between elements in the scene less visually obvious. The transition effect 121 may include, but is not limited to: a cross-fade with dimming in the transition area, the introduction of a fuzzy boundary ring, and radial gradient dimming of the entire surrounding scene (brightest at the edges and gradually darker towards the center).

[0069] As mentioned above, in Figure 3 In the example of , two independent scenes within the virtual reality simulation are maintained. The rendered first scene 113 includes a display of the main virtual reality simulation 115, and the camera used to render it may be affected by input control (e.g., a game controller, keyboard, or mouse) separate from the display device 106 (e.g., a head-mounted display) or the logic of the simulation itself. When no camera movement is applied that is independent of the movement provided by head tracking (e.g., head movement information 14 received from the position tracking system 107), the rendered first scene 113 can occupy the entire display. However, once independent movement is detected that is inconsistent with vestibular signals, the display area of ​​the main scene (e.g., the rendered first scene 113) can be dynamically reduced so that the scene is pulled into the visual center and then surrounded by a secondary scene (e.g., the rendered second scene 114) that occupies the user's peripheral vision.

[0070] A rendered second scene 114 (e.g., a peripheral scene) is rendered using a camera controlled only by head tracking motion (e.g., head motion information 14 received from a position tracking system 107) to match the vestibular sensory input. The peripheral scene provides the visual cues 17 necessary to suppress motion sickness. The elements of this scene may have a close relationship with the content from the main scene, or they may be composed of a completely separate collection of media. Examples of peripheral scene compositions include: a virtual lobby area containing art assets from a game that includes the main scene, a virtual theater environment, or even a display of the user's current physical environment (implemented by spatial reconstruction, reprojected feeds from external camera(s) present on the HMD, or a transparent view of the real world for an AR-enabled display). In some implementations, the peripheral scene is constructed in such a way that strong stereoscopic visual cues with detailed geometry and texture structure are provided to the user.

[0071] As described in detail above, the camera control of both scenes can be described in terms of the matrices used in their rendering.

[0072] After all viewer motion not derived from head tracking has ceased for a predetermined period of time, the rendered first scene 114 (e.g., the main scene) can be dynamically expanded back to cover the entire display. The animated expansion of the inner saucer area is preferred to the immediate capture of the full view. The main scene can then remain fully covered until another auxiliary independent motion effect is detected, and the rendered second scene 114 (e.g., the peripheral scene) is made visible again. It may be desirable to make this transition from the full scene to the dual scene more direct to provide immediate anchoring cues. The relative diameters of the rendered first scene 113 and the rendered second scene 114 (e.g., the main scene and the peripheral scene, respectively) can depend in part on the field of view of the display, and can even be controlled by the user based on the individual's susceptibility to motion sickness.

[0073] Reference now Figure 4 , the method 400 for presenting a VR image may be performed on the computer device 102 ( Figure 1 ) operating system 110( Figure 1 For example, the method 400 may be used to display frames of a virtual reality simulation 22 generated by a computer device 102 on a display device 106 (e.g., an HMD) that reduces and / or prevents discomfort, such as simulation sickness or motion sickness, for a user when viewing the presented frames.

[0074] At 402, method 400 may include receiving head motion information from a position tracking system that tracks the position and orientation of a user's head. For example, application(s) 10( Figure 1 ) and / or operating system 110 ( Figure 1 ) can receive head movement information 14 from the position tracking system 107 ( Figure 1 ), the position tracking system 107 can track the position and orientation of the user's head and / or display device 106 (e.g., an HMD that can be worn by the user), where the user is looking (e.g., the gaze direction of the eyes), and / or the user's focus. The head movement information 14 can correspond to the user's movement in the real world.

[0075] At 404, method 400 may include receiving virtual motion information independent of head motion information for use in virtual reality simulation. For example, application(s) 10( Figure 1 ) and / or operating system 110 ( Figure 1 ) can be from one or more external controllers 108 ( Figure 1 ) receiving virtual motion information 12( Figure 1 ). The external controller 108 may include, but is not limited to, a game controller, a joystick, a keyboard, and a mouse. Additionally, the virtual motion information 12 may be received based on, for example, the design of the virtual reality simulation 22.

[0076] At 406, method 400 may include generating a first scene of the virtual reality simulation based on the virtual motion information and the head motion information. A first application of one or more applications 10 in combination with GPU 24 may generate a first scene 16 of virtual reality simulation 22 based on virtual motion information 12 and head motion information 14. First scene 16 may correspond to master virtual reality simulation 115 of virtual world 11 and may have a motion volume that tracks the motion of virtual world 11.

[0077] At 408, method 400 may include generating a set of visual cues based on the head motion information. For example, a first application or a second application in one or more applications 10 or an operating system 110 may generate one or more visual cues 17 based on the received head motion information 14. The visual cues 17 and the relative motion of the visual cues 17 may be generated based on the head motion information 14 received from a position tracking system 107, which may monitor the position, orientation, and movement of a display device 106 (e.g., an HMD). The visual cues 17 may include spatial elements with detailed texture structures to help minimize and / or suppress motion sickness of the user. For example, the visual cues 17 may include, but are not limited to: lines or abstract design patterns, lights, sparks, smoke, snowflakes, bubbles, and particle effects. The texture of the visual cues 17 may depend on the content of the first scene 16 of the virtual reality simulation 22, or the visual cues 17 may consist of an entire separate media set. Additionally, the visual cues 17 may generate visual signals to the brain that may be consistent with visual signals generated by the vestibular system of a user (eg, wearing an HMD) in response to the user's physical movements, without distracting the user from the virtual simulation 22 .

[0078] At 410, the method 400 may optionally include generating a second scene of the virtual reality simulation having a set of visual cues. For example, the application(s) 10 and / or the operating system 110 may also generate the second scene 20, for example based on the running auxiliary virtual reality simulation 117, based on the head movement information 14, to be inserted into or combined with the first scene 16 of the virtual reality simulation 22. The second scene 20 may include visual cues 17 that match the vestibular sensory information of the user. In this way, the second scene 20 may provide the visual cues 17 necessary to suppress and / or prevent the user's motion sickness. The second scene 20, for example, based on the running auxiliary virtual reality simulation 117, may correspond to a different virtual scene (compared to the first scene 16) or to an independent view of an area of ​​the original virtual scene (e.g., the first scene 16). The elements included in the second scene 20 may have a close relationship with the content from the first scene 16, or the elements may be composed of a completely separate set of media. The second scene 20 can be constructed in such a way that strong stereoscopic visual cues 17 with detailed geometric and textural structures are provided to the user to help minimize and / or suppress motion sickness of the user. For example, for a virtual lobby scene, the visual cues 17 may include, but are not limited to, shelves, lines or designs on the walls, lamps, and furniture. In another example, if the second scene including the visual cues 17 is selected as an outdoor recess in a forest, the details of the visual cues 17 may include trees, leaves, and grass. The geometry and texture of the visual cues 17 depend on the selected scene. The second scene 20 may include an amount of motion that is independent of the first scene 16, where the independent motion only tracks motion sensed by the user's inner ear. For example, the amount of motion of the second scene 20 is based on head motion information 14 from the position tracking system 107, which monitors the movement of the user's head and / or the movement of the display device 106 (e.g., HMD).

[0079] At 412, method 400 may include rendering a frame including a set of visual cues (or, optionally, a second scene) and a first scene of a virtual reality simulation. For example, a first application of one or more applications 10 (if both the primary virtual reality simulation 115 and the secondary virtual reality simulation 117 are running) may combine the visual cues 17 (or the second scene 20) and the first scene 16, and optionally forward the combination to a compositor 18. Alternatively, for example, the compositor 18 may receive the visual cues 17 (or the second scene 20) from the first application or the second application of the applications 10 or from the operating system 110 separately, and receive the first scene 16 from the first application of the applications 10. The compositor 18 may operate in conjunction with the GPU 24 to combine the set of visual cues 17 (or the second scene 20; or the rendered visual cues or the second scene 114) and the first scene 16 (or the rendered first scene 113) to obtain a display frame 25 representing a scene in the virtual reality simulation 22. The visual cues 17 may be placed anywhere within the viewing area of ​​the display frame 25 presented on the display panel 32 of the display device 106. In one implementation, the compositor 18 may combine the visual cues 17 (or rendered visual cues) within the first scene 16 (or rendered first scene 113), for example as a grid of visual cues 17 (or rendered visual cues) blended across the entire first scene 16 (or rendered first scene 113). In another implementation, the compositor 18 may combine the second scene 20 (or rendered visual cues or second scene 114) in a peripheral area around the first scene 16 (or rendered first scene 113).

[0080] At 414, method 400 may include sending a frame of the virtual reality simulation for presentation on a display device. For example, compositor 18 and / or GPU 24 may generate display frame 25 and may operate in cooperation with display controller 26 to determine when to send display frame 25 for presentation on display device 106, display frame 25 including rendered first scene 113 and rendered visual cues or second scene 114 and optionally including transition effects 121. Compositor 18 and / or GPU 24 may be capable of generating new rendered display frames 25 for virtual reality simulation 22 at a refresh rate of display device 106. For example, the refresh rate for virtual reality may be 90 Hz, resulting in a refresh every 11 milliseconds.

[0081] In this way, the rendered display frame 25 presented on the display device 106 may include a rendered first scene 113 with (multiple) visual cues 17, or a rendered first scene 113 and a rendered second scene 114 (which includes (multiple) visual cues 17). In some implementations, the rendered first scene 113 corresponds to a central area or a focal area that the user is viewing, and the rendered first scene 113 may be based on the main virtual reality simulation 115 and may have an amount of virtual motion corresponding to the virtual motion information 12. The rendered second scene 114 may be located in a perimeter around the first scene 113, for example, in an area outside the focal area of ​​the rendered first scene 113. In addition, the (multiple) visual cues 17 and / or the rendered second scene 114 have independent motion, for example, relative to the motion of the rendered first scene 113, wherein the independent motion is only associated with the head motion information 14. The rendered display frame 25 may also include a transition effect 121 between the rendered first scene 113 and the visual cues 17 and / or the rendered second scene 114. The transition effect 121 can make the discontinuity between the rendered first scene 113 and the elements in the visual cue 17 and / or the rendered second scene 114 less visually obvious. Therefore, the display frame 25 of the virtual reality simulation 22 having one area associated with virtual motion and another area associated only with head motion, as generated by the method and apparatus of the present disclosure, can improve user comfort by reducing or eliminating conflicting signals from the user's eyes and inner ears, thereby reducing or eliminating nausea, motion sickness, or vertigo.

[0082] Reference now Figure 5 , illustrates a method that can be used by a computer device 102 to generate and display a virtual world 11 ( Figure 1 )'s virtual reality simulation 22( Figure 1 ), the rendered display frame 25 being composed of the rendered first scene 113 ( Figure 1 ) and the rendered second scene 114 ( Figure 1 ) is defined as follows: the rendered first scene 113 has a motion based at least on the virtual motion information 12, and the rendered second scene 114 has a motion based only on the received head motion information 14 around the rendered first scene 113. When watching the virtual reality simulation 22, since the motion of the rendered second scene 114 is similar to the motion sensed by the vestibular system of the user, presenting the display frame 25 on the display device 106 (e.g., HMD) can minimize and / or prevent user discomfort (e.g., simulation sickness or motion sickness).

[0083] The illustrated implementation of the method 500 includes a first application in one or more applications generating a first scene 16, and an operating system 110 generating a second scene 20. However, as described above, other implementations may utilize a first application in one or more applications 10 or a second application in one or more applications 10 to generate a second scene. Similar actions as discussed below will apply to those other implementations.

[0084] At 502 and 504, method 500 may include transforming the received virtual motion information and the received head motion information into virtual coordinates. The original physical coordinates received for the virtual motion information 12 and / or the head motion information 14 may be transformed into a position and orientation as represented in a simulated coordinate system. The first application in the applications 10 may transform the virtual motion information 12 and / or the head motion information 14 into associated character motion in the virtual world 11. For example, a first application in the applications 10 may convert the received head movement information 14 and / or the received virtual movement information 12 into a possibly modified “game space” pose established during VR device setup (e.g., modifying the origin / center of the space from an origin / center specified by an outside-in tracking system using absolute sensor positioning to an origin / center specified by a user when selecting the center of the room based on furniture or other objects in the room), and then potentially into a virtual user pose in a simulation (e.g., a tracking system may maintain coordinates in meters, while a game may establish its own coordinate system in feet, and a tracking system may also treat the forward direction Z as a positive value, while a simulation may use negative Z values ​​to express forward).

[0085] At 506, method 500 may include combining the virtual coordinates of the virtual motion information and the head motion information to determine the position of the character in the virtual world. For example, a first application in application 10 may use a first transformation matrix (e.g., W*V as described above) to transform the character into a virtual world. simulation *V head *P) to combine virtual head movement information 504 and virtual movement information 502 to determine the virtual position of the character and / or object in the virtual world 11. A first application in the one or more applications 10 may first transform the model geometry into world space, then transform into the virtual position and orientation of the user in the simulation, followed by the relative repositioning / reorientation specified by the head tracked gesture (e.g., head movement information 14).

[0086] At 508 and 510, method 500 may include rendering and outputting the first scene image frame. The first application in application 10 may generate a first scene 16 ( Figure 1). In one implementation, the first application in the applications 10 may generate the first scene 16 based on the running main virtual reality simulation 115. The first scene 16 may correspond to the original view area of ​​the virtual scene and may track the movement of the virtual scene. The first application in the applications 10 may also output a projection of the first scene 16 for display. For example, the first application in the applications 10 may send the first scene 16 to the GPU 24 to render the first scene 16, thereby outputting a rendered first scene 113.

[0087] At 512, the method 500 may include transforming the received head motion information into anchor environment coordinates. For example, the operating system 110 may use a second transformation matrix (e.g., W*V as described above) when rendering each eye for the second scene 20. head *P).

[0088] At 514 and 516, the method 500 may include rendering and outputting the second scene image frame. The operating system 110 may render the second scene 20 for the anchoring environment around the first scene 16 based on the head motion information 14. The second scene 20 may be based on the auxiliary virtual reality simulation 117, and may correspond to a different virtual scene or an independent view corresponding to the original virtual scene area. The elements included in the second scene 20 may have a close relationship with the content from the first scene 16, or the elements may be composed of a completely independent media collection. The second scene 20 may have an amount of motion that is independent of the motion of the virtual scene (e.g., the first scene 16). Specifically, the surrounding area of ​​the anchoring environment may have an amount of motion that tracks the motion sensed by the user's inner ear (e.g., an amount of motion based only on the head motion information 14 from the position tracking system 107). In one implementation, the second scene 20 includes a visual cue 17 or is in the form of a visual cue 17. In addition, for example, the operating system 110 may send the second scene 20 to the GPU 24 to render the second scene 20, thereby outputting the rendered visual cue or the second scene 114.

[0089] At 518, method 500 may include combining the rendered second scene image frame and the rendered first scene image frame, and outputting the combined display image for display. The synthesizer 18 may optionally reproject the rendered first scene 113 and / or the rendered visual clue or the second scene 114 based on the updated head movement information 14. The synthesizer 18 may generate a combined image frame (e.g., display frame 25) by combining the rendered first scene 113 and the rendered visual clue or the second scene 114. The combined image frame may include a transition effect 121 between the rendered first scene 113 and the rendered visual clue or the second scene 114. The transition effect 121 may make the discontinuity between the rendered first scene 113 and the rendered visual clue or the second scene 114 less visually obvious. The synthesizer 18 may output a combined image frame (e.g., display frame 25) for display on the display device 106.

[0090] At 520, method 500 may include displaying a combined virtual image. The displayed combined virtual image (e.g., display frame) may include a rendered first scene 113 having a first display area, the first display area having a motion amount that tracks the motion of the virtual scene (e.g., corresponding to virtual motion information 12). At the same time, the rendered second scene 114 may be at the periphery of the rendered first scene 113 and may have a second display area that is different from the first display area. The peripheral area occupied by the rendered second scene 114 in the display frame 25 may have a motion amount that is independent of the motion of the virtual scene (e.g., the rendered first scene 113). Specifically, the peripheral area may have a motion amount that tracks the motion sensed by the user's inner ear, for example, based on the head motion information 14 from the position tracking system 107. In this way, the combined image of the display frame 25 can improve user comfort by reducing or eliminating conflicting signals from the user's eyes and inner ears, thereby reducing or eliminating nausea, motion sickness, or vertigo.

[0091] Reference now Figure 6 , illustrates an example of a computer device 102 according to an implementation, and Figure 1 In contrast, computer device 102 includes additional component details.

[0092] The computer device 102 may include a communication component 40 that is provided for establishing and maintaining communications with one or more parties using hardware, software, and services as described herein. The communication component 40 may carry communications between components on the computer device 102 and between the computer device 102 and external devices (e.g., devices located on a communication network and / or devices connected serially or locally to the computer device 102). For example, the communication component 40 may include one or more buses and may also include a transmit chain component and a receive chain component associated with a transmitter and a receiver, respectively, operable to interface with external devices.

[0093] Additionally, computer device 102 may include a data repository 42, which may be any suitable combination of hardware and / or software that provides mass storage of information, databases, and programs used in conjunction with the implementations described herein. For example, data repository 42 may be a database for application 10 ( Figure 1 )、Synthesizer 18( Figure 1 )、GPU 24( Figure 1 )、Display controller 26( Figure 1 ) and / or display interface 28 ( Figure 1 ) data repository.

[0094] The computer device 102 may also include a user interface component 44, which is operable to receive input from a user of the computer device 102 and is also operable to generate output for presentation to the user. The user interface component 44 may include one or more input devices, including but not limited to a keyboard, a numeric keypad, a mouse, a touch-sensitive display, a navigation key, a function key, a microphone, a speech recognition component, any other mechanism capable of receiving input from a user, or any combination thereof. In addition, the user interface component 44 may include one or more output devices, including but not limited to a display, a speaker, a tactile feedback mechanism, a printer, any other mechanism capable of presenting output to a user, or any combination thereof.

[0095] In one implementation, user interface component 44 may send and / or receive messages corresponding to the operation of application 10, compositor 18, GPU 24, display controller 26, and / or display interface 28. In addition, processor 36 executes application 10, compositor 18, GPU 24, display controller 26, and / or display interface 28, and memory 38 or data repository 42 may store them.

[0096] As used in this application, the terms "component", "system" etc. are intended to include computer-related entities, such as but not limited to hardware, firmware, a combination of hardware and software, software, or software in execution. For example, a component can be but not limited to a process, a processor, an object, an executable file, a thread of execution, a program and / or a computer running on a processor. By way of illustration, both the application and the computer device running on a computer device can be a component. One or more components can reside in a process and / or a thread of execution, and a component can be located on a computer and / or distributed between two or more computers. Additionally, these components can be executed from various computer-readable media having various data structures stored thereon. A component can, for example, communicate and / or communicate with other systems by means of a signal according to a signal having one or more data packets (for example, data from a component interacting with another component of a local system, a distributed system and / or across a network (for example, the Internet)).

[0097] Furthermore, the term "or" is intended to mean an inclusive "or" rather than an exclusive "or." That is, unless specified otherwise or clear from the context, the phrase "X employs A or B" is intended to mean any of the natural inclusive permutations. That is, the phrase "X employs A or B" is satisfied by any of the following: X employs A; X employs B; or X employs A and B. Additionally, with respect to the singular, the articles "a" and "an" as used in this application and the appended claims should generally be construed to mean "one or more" unless specified otherwise or clear from the context.

[0098] Various implementations or features may be presented in terms of systems that may include several devices, components, modules, etc. It should be understood and appreciated that the various systems may include additional devices, components, modules, etc. and / or may not include all of the devices, components, modules, etc. discussed in conjunction with the figures. Combinations of these methods may also be used.

[0099] Various exemplary logics, logic blocks and actions of the methods described in conjunction with the embodiments disclosed herein may be implemented or performed using a general purpose processor, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic device, discrete gate or transistor logic, one of discrete hardware components specifically programmed or any combination thereof designed to perform the functions described herein. A general purpose processor may be a microprocessor, but alternatively, the processor may be any conventional processor, controller, microcontroller or state machine. The processor may also be implemented as a combination of computer devices (e.g., a combination of a DSP and a microprocessor), multiple microprocessors, one or more microprocessors in combination with a DSP core or any other such configuration. Additionally, at least one processor may include one or more components operable to perform one or more of the above steps and / or actions.

[0100] In addition, the steps and / or actions of the method or algorithm described in conjunction with the implementation disclosed herein may be directly embodied in hardware, in a software module executed by a processor, or in a combination of the two. The software module may reside in a RAM memory, a flash memory, a ROM memory, an EPROM memory, an EEPROM memory, a register, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art. An exemplary storage medium may be coupled to a processor so that the processor can read information from the storage medium and write information to the storage medium. In an alternative, the storage medium may be an integral part of the processor. In addition, in some implementations, the processor and the storage medium may reside in an ASIC. Additionally, the ASIC may reside in a user terminal. In an alternative, the processor and the storage medium may reside in a user terminal as discrete components. Additionally, in some implementations, the steps and / or actions of the method or algorithm may reside as one or any combination or set of codes and / or instructions on a machine-readable medium and / or a computer-readable medium that may be incorporated into a computer program product.

[0101] In one or more implementations, the functions described may be implemented in hardware, software, firmware, or any combination thereof. If implemented in software, the functions may be stored or transferred to a computer-readable medium as one or more instructions or codes. Computer-readable media include computer storage media and communication media, and communication media include any media that facilitates the transfer of computer programs from one place to another. The storage medium may be any available medium that can be accessed by a computer. As an example and not limitation, such a computer-readable medium may include RAM, ROM, EEPROM, CD-ROM or other optical disk storage devices, magnetic disk storage devices or other magnetic storage devices, or any other medium that can be used to store the desired program code in the form of an instruction or data structure and can be accessed by a computer. The disks and optical disks used herein include optical disks (CDs), laser optical disks, optical disks, digital versatile disks (DVDs), floppy disks, and blue-ray disks, wherein the disks typically reproduce data magnetically, and the optical disks typically reproduce data optically with lasers. The above combination should also be included in the scope of computer-readable media.

[0102] Although the implementation of the present disclosure has been described in conjunction with its examples, it will be appreciated by those skilled in the art that changes and modifications may be made to the above implementations without departing from the scope of the present invention. Other implementations will be apparent to those skilled in the art by consideration of the specification or practice of the examples disclosed herein.

Claims

1. A computer device comprising: a memory storing at least one application configured to run a virtual reality simulation; a processor in communication with the memory and configured to execute the at least one application to run the virtual reality simulation; as well as an operating system in communication with the memory and the processor, wherein the operating system, the processor, and the at least one application are operable to: receiving head movement information from a position tracking system that tracks the position and orientation of a user's head; receiving virtual motion information independent of the head motion information for use in virtual reality simulation; generating a first scene of the virtual reality simulation based at least on the virtual motion information and the head motion information; generating a set of visual cues on a set of frames, the set of visual cues having an amount of motion corresponding to the head motion information, wherein the set of visual cues includes geometric and texture structures for insertion into the virtual reality simulation; Rendering the frame set, the frame set including the first scene of the virtual reality simulation and a second scene independent of the first scene for the virtual reality simulation, wherein the second scene includes the set of visual cues, and the virtual reality simulation includes a transition effect between the first scene and the second scene and a transition between displaying the first scene and displaying both the first scene and the second scene based on whether the virtual motion information is received; as well as The set of frames of the virtual reality simulation is sent for presentation on a display device.

2. The computer device of claim 1 , wherein the operating system, the processor, and the at least one application are further operable to: remove the set of visual cues from the virtual reality simulation when all sources of motion input other than head movement have ceased for a period of time.

3. The computer device of claim 2, wherein the operating system, the processor, and the at least one application are further operable to: upon receiving a motion input source other than the head motion information, reinsert the set of visual cues into the virtual reality simulation. The computer device according to claim 1 , wherein the second scene is a surrounding scene around the first scene.

5. A computer device according to claim 4, wherein the operating system, the processor and the at least one application are further operable to: dynamically expand the first scene to cover an entire viewing area of ​​the display device, and remove the second scene from the viewing area when all motion input sources other than the head motion information have stopped for a period of time.

6. A computer device according to claim 5, wherein the operating system, the processor and the at least one application are further operable to: dynamically reduce the size of the view area of ​​the first scene, and reinsert the second scene with the set of visual cues into the virtual reality simulation upon receiving motion input independent of the head motion information.

7. The computer device of claim 1, wherein the operating system, the processor, and the at least one application are further operable to: receiving video information from one or more video cameras; Visual cues based on the motion input are formed based on the video information or based on an environment reconstruction based on the video information.

8. A method for presenting a virtual reality image, comprising: receiving head movement information from a position tracking system that tracks the position and orientation of a user's head; receiving virtual motion information independent of the head motion information for use in virtual reality simulation; generating a first scene of the virtual reality simulation based at least on the virtual motion information and the head motion information; generating a set of visual cues on a set of frames, the set of visual cues having an amount of motion corresponding to the head motion information, wherein the set of visual cues includes geometric and texture structures for insertion into the virtual reality simulation; rendering, at an operating system executed on a computer device, the frame set, the frame set including the first scene of the virtual reality simulation and a second scene independent of the first scene for the virtual reality simulation, wherein the second scene includes the set of visual cues, and the virtual reality simulation includes a transition effect between the first scene and the second scene and a transition between displaying the first scene and displaying both the first scene and the second scene based on whether the virtual motion information is received; as well as The set of frames of the virtual reality simulation is sent for presentation on a display device.

9. The method according to claim 8, further comprising: The set of visual cues is removed from the virtual reality simulation when all sources of motion input other than head motion have ceased for a period of time.

10. The method according to claim 9, further comprising: Upon receiving a source of motion input other than the head motion information, the set of visual cues is reinserted into the virtual reality simulation. The method according to claim 8 , wherein the second scene is a surrounding scene around the first scene.

12. The method according to claim 11, further comprising: dynamically expanding the first scene to cover an entire viewing area of ​​the display device; as well as When all sources of motion except the head motion information have ceased for a period of time, the second scene is removed from the viewing area.

13. The method according to claim 12, further comprising: The size of the viewing area of ​​the first scene is dynamically reduced, and upon receiving motion input independent of the head motion information, the second scene with the set of visual cues is reinserted into the virtual reality simulation.

14. The method according to claim 8, further comprising: receiving video information from one or more video cameras; Visual cues based on the motion input are formed based on the video information or based on an environment reconstruction based on the video information.

15. A computer-readable medium storing instructions executable by a computer device, comprising: at least one instruction for causing the computer device to receive head movement information from a position tracking system that tracks the position and orientation of a user's head; at least one instruction for causing the computer device to receive virtual motion information independent of the head motion information for use in a virtual reality simulation; at least one instruction for causing the computer device to generate a first scene of the virtual reality simulation based at least on the virtual motion information and the head motion information; at least one instruction for causing the computer device to generate a set of visual cues over a set of frames, the set of visual cues having an amount of motion corresponding to the head motion information, wherein the set of visual cues includes geometric and textural structures to be inserted into the virtual reality simulation; at least one instruction for causing the computer device to render the set of frames, the set of frames comprising the first scene of the virtual reality simulation and a second scene independent of the first scene for the virtual reality simulation, wherein the second scene comprises the set of visual cues, and the virtual reality simulation comprises a transition effect between the first scene and the second scene and a transition between display of the first scene and display of both the first scene and the second scene based on whether the virtual motion information is received; as well as At least one instruction for causing the computer device to send the set of frames of the virtual reality simulation for presentation on a display device.

Images