Asynchronous virtual reality interaction

Abstract

A method is provided, comprising: recording game plot metadata generated from the execution of a first session of a video game, the execution of the first session being driven by a first user using a first head-up display (HMD) through an interactive game plot of the video game, wherein the execution of the first session renders a first view of a virtual environment, the first view being derived from a first position determined by the interactive game plot and based on tracked movement of the first HMD; transmitting the game plot metadata to a client device; tracking movement of a second HMD; and executing a second session of the video game via the client device using the game plot metadata to reconstruct game plot events, wherein the execution of the second session renders a second view of the virtual environment, the second view being derived from a second position determined based on the first position and based on tracked movement of the second HMD.

Description

[0001] This application is a divisional application of the invention patent application with application number 201910107402.4, application date February 2, 2019, and invention title "Asynchronous Virtual Reality Interaction". Technical Field

[0002] This disclosure relates to devices and methods for providing asynchronous virtual reality interaction. Background Technology

[0003] The video game industry has changed significantly over the years. As computing power has expanded, video game developers have also created game software that leverages this increased computing power. To this end, video game developers have been writing games that incorporate complex mechanics and mathematical calculations to produce highly detailed and engaging gaming experiences.

[0004] Example gaming platforms include the Sony PlayStation®, Sony PlayStation 2® (PS2), Sony PlayStation 3® (PS3), and Sony PlayStation 4® (PS4), each sold as a game console. Game consoles are known to be designed to connect to a monitor (usually a television) and enable user interaction via handheld controllers. Game consoles are designed with specialized processing hardware, including a CPU, a graphics synthesizer for handling intensive graphics operations, vector units for performing geometric transformations, and other glued hardware, firmware, and software. Game consoles may also be designed with a disc reader for receiving game discs for local gaming. Online gaming is also possible, where users can interactively play against or play together with other users via the internet. As the complexity of games continues to pique player interest, game and hardware manufacturers are constantly innovating to enable additional interactivity and computer programs.

[0005] A growing trend in the computer gaming industry is the development of games that increase interaction between the user and the game system. One way to achieve a richer interactive experience is by using a wireless game controller. The game system tracks the movement of the wireless game controller to track the player's movements and use those movements as input for the game. Generally speaking, gesture input refers to enabling electronic devices, such as computing systems, video game consoles, and smart devices, to respond to certain gestures made by the player and captured by the electronic device.

[0006] Another way to achieve a more immersive and interactive experience is by using a head-up display (HMD). An HMD is worn by the user and can be configured to display various graphics, such as views of virtual spaces / environments. The graphics displayed on a head-up display can cover most or even all of the user's field of vision. Therefore, an HMD can provide the user with a visually immersive experience. Using an HMD to experience a virtual environment in this way is often called virtual reality (VR), hence HMDs are also known as VR headsets.

[0007] Another industry trend involves the development of cloud-based gaming systems. These systems may include a remote processing server that executes the game application and communicates with a local thin client, which may be configured to receive input from the user and render video on a display. In some implementations, the remote processing server may include the physical hardware of a game console, or such hardware that replicates the physical hardware of a game console. In other implementations, the remote processing server may define a virtual machine that emulates the hardware of a game console.

[0008] It is against this backdrop that the implementation plan disclosed herein came into being. Summary of the Invention

[0009] Embodiments of this disclosure include methods and systems for providing asynchronous virtual reality interaction.

[0010] In some implementations, a method is provided that includes the following operations: recording game plot metadata generated from the execution of a first session of a video game, the execution of the first session being driven by a first user using a first head-up display (HMD) through an interactive game plot of the video game, wherein the execution of the first session renders a first view of a virtual environment of the video game to be presented via the first HMD, the first view being derived from a first position in the virtual environment determined by the interactive game plot, the first view also being based on tracked movement of the first HMD; after the first session is completed, transmitting the game plot metadata to a client device; tracking movement of a second HMD via the client device; and executing a second session of the video game via the client device using the game plot metadata to reconstruct game plot events from the first session in the second session, wherein the execution of the second session renders a second view of the virtual environment to be presented via a second HMD, the second view being derived from a second position in the virtual environment determined based on the first position in the virtual environment, the second view also being based on tracked movement of the second HMD.

[0011] In some implementations, the tracking movement of the first HMD includes the tracking orientation of the first HMD in a first local environment in which the first HMD is placed; wherein the orientation of the first view in the virtual environment is determined by the tracking orientation of the first HMD.

[0012] In some implementations, the tracking movement of the second HMD includes the tracking orientation of the second HMD in a second local environment in which the second HMD is placed; wherein the orientation of the second view in the virtual environment is determined by the tracking orientation of the second HMD.

[0013] In some implementations, the first position in the virtual environment is a predefined first position placed in a virtual vehicle within the virtual environment; wherein the second position in the virtual environment is a predefined second position in the virtual vehicle.

[0014] In some implementations, the predefined first position in the virtual vehicle is the driver's position in the virtual vehicle; and the predefined second position in the virtual vehicle is the passenger's position in the virtual vehicle.

[0015] In some implementations, the game plot metadata includes game state values ​​generated by the execution of the first session of the video game.

[0016] In some implementations, the execution of the first session includes processing input data generated by the first user from the interactive game scenario; wherein the game scenario metadata includes the input data.

[0017] In some implementations, the input data is generated via an input device operated by the first user.

[0018] In some implementations, the first session is performed by a computing device remote from the client device, the computing device and the client device being connected to a network, through which the game plot metadata is transmitted.

[0019] In some implementations, a method is provided that includes: recording game plot metadata generated from the execution of a first session of a video game, the execution of which is driven by a first user using a first head-up display (HMD) through an interactive game plot of the video game, wherein the execution of the first session renders a first view of a virtual environment of the video game to be presented via the first HMD, the first view being derived from a first position in the virtual environment determined by the interactive game plot, the first view also being based on tracked movement of the first HMD; after the first session is completed, transmitting the game plot metadata to a client device; tracking movement of a second HMD via the client device; and executing a second session of the video game via the client device using the game plot metadata to reconstruct game plot events from the first session in the second session, wherein the execution of the second session renders a second view of the virtual environment to be presented via a second HMD, the second view being derived from a second position in the virtual environment determined based on the tracked movement of the second HMD.

[0020] In some implementations, the second location is further determined using input data generated by a second user from the interactivity of the second session with the video game.

[0021] In some implementations, the input data is generated via an input device operated by the second user.

[0022] In some implementations, the tracking movement of the second HMD includes the tracking orientation of the second HMD in a local environment in which the second HMD is placed; wherein the orientation of the second view in the virtual environment is determined by the tracking orientation of the second HMD.

[0023] In some implementations, the rendering of the second view is configured to have settings that adjust based on the orientation of the second view relative to the first position of the first view in the virtual environment.

[0024] In some implementations, the adjustment of the settings includes adjusting the level of detail of the rendering of the second view, such that the level of detail is increased when the orientation of the second view is toward the first position of the first view, and decreased when the orientation of the second view is away from the first position of the first view.

[0025] In some implementations, the level of detail is defined by one or more of the following: the amount of virtual objects, the amount of color saturation, the amount of texture, the amount of shadows, the resolution level, and the graphics complexity.

[0026] Other aspects and advantages of this disclosure will become apparent from the following detailed description taken in conjunction with the accompanying drawings, which illustrate the principles of this disclosure by way of example. Attached Figure Description

[0027] This disclosure can be better understood by referring to the following description taken in conjunction with the accompanying drawings, in which:

[0028] Figure 1 A system for an interactive game plot for a video game according to an embodiment of the present disclosure is shown.

[0029] Figure 2 A head-up display (HMD) according to an embodiment of the present disclosure is shown.

[0030] Figure 3 The functionality of an HMD combined with a video game being played is conceptually illustrated according to an embodiment of this disclosure.

[0031] Figure 4 A conceptual illustration shows asynchronous interaction between users of a head-up display according to an embodiment of the present disclosure.

[0032] Figure 5 The present invention conceptually illustrates a change in the view orientation of a first user during a game scenario session according to an embodiment of the present disclosure, and a change in the view orientation of a second user when watching a replay of the first user's game scenario session.

[0033] Figure 6A A conceptual top view of a virtual character in a virtual environment according to an embodiment of the present disclosure is shown, illustrating the relationship between the view position and view orientation in the virtual environment.

[0034] Figure 6B The graph shown illustrates the horizontal angular orientation of the virtual character 408 (reference 604) and the angular position of the virtual character 426 relative to the angular orientation of the virtual character 408 according to an embodiment of the present disclosure (reference 606).

[0035] Figure 7 The concept illustrates that a second user 420, according to an embodiment of the present disclosure, is able to navigate and watch independently of the first user when viewing a replay of the first user's game plot in a virtual environment.

[0036] Figure 8 A conceptual illustration shows a virtual character in a virtual environment during the playback of a game plot for a first user 400 under rendering adjustments, according to an embodiment of the present disclosure.

[0037] Figure 9A conceptual illustration shows a virtual character in a virtual environment during the playback of a game plot for a first user 400 under rendering adjustments, according to an embodiment of the present disclosure.

[0038] Figure 10 The conceptual illustration shows the path traversed by the virtual character of a first user 400 according to an embodiment of the present disclosure, and the path traversed by the virtual character of a second user 420 which is kept near the virtual character of the first user 400.

[0039] Figure 11 The present invention conceptually illustrates a virtual character in a virtual environment according to an embodiment of the present disclosure, and variations in the areas in which the virtual character may be located.

[0040] Figure 12 A system for implementing asynchronous interaction between HMD users is shown according to an embodiment of the present disclosure.

[0041] Figure 13 The components of a head-up display according to an embodiment of the present disclosure are shown.

[0042] Figure 14 This is a block diagram of a game system 1400 according to various embodiments of this disclosure. Detailed Implementation

[0043] The following embodiments of this disclosure provide methods and systems for providing asynchronous virtual reality (VR) interaction. Currently, the virtual reality market is expanding, but it is not very large compared to the much larger video game market. Therefore, it may be difficult to find other virtual reality users online for synchronous interaction. Therefore, providing asynchronous interaction utilizing a network of virtual reality users is beneficial.

[0044] However, it will be apparent to those skilled in the art that this disclosure can be practiced without some or all of these specific details. In other instances, well-known procedures have not been described in detail to avoid unnecessarily obscuring this disclosure.

[0045] Figure 1A system for an interactive game scenario for a video game, according to an embodiment of this disclosure, is shown. User 100 is shown wearing a head-up display (HMD) 102, also known as a virtual reality (VR) headset. The HMD 102 is worn in a manner similar to glasses, goggles, or a helmet and is configured to display a video game or other content to user 100. The HMD 102 provides a highly immersive experience to the user by providing a display mechanism close to the user's eyes. Therefore, the HMD 102 can provide a display area to each of the user's eyes, which occupies most or even the entirety of the user's field of vision. The term "virtual reality" generally refers to viewing a virtual space / environment through an HMD, such that the view of the virtual space displayed by the HMD to the user responds in real time to the tracking movement of the HMD, thereby providing the user with the feeling of physical presence in the virtual space / environment. For example, when the user moves their head in a given direction, the view displayed through the HMD is updated to display the view in that direction in the virtual space.

[0046] In one embodiment, HMD 102 can be connected to computer 106. The connection to computer 106 can be wired or wireless. Computer 106 can be any general-purpose or special-purpose computer known in the art, including but not limited to game consoles, personal computers, laptops, tablets, mobile devices, cellular phones, tablet computers, thin clients, set-top boxes, media streaming devices, etc. In one embodiment, computer 106 can be configured to execute a video game and output video and audio from the video game for rendering by HMD 102.

[0047] User 100 can operate interface object 104 (e.g., controller device, glove controller, etc.) to provide input for the video game. Additionally, camera 108 can be configured to capture images of the interactive environment in which user 100 is located. These captured images can be analyzed to determine the position and movement of user 100, HMD 102, and interface object 104. In one embodiment, interface object 104 includes lights that can be tracked to determine its position and orientation. Additionally, HMD 102 may include one or more lights that can be tracked to determine the position and orientation of HMD 102. Camera 108 may include one or more microphones to capture sound from the interactive environment. The sound captured by the microphone array can be processed to identify the location of sound sources. Sound from the identified locations can be selectively utilized or processed to exclude other sounds not originating from the identified locations. Furthermore, camera 108 can be defined to include multiple image capture devices (e.g., a stereo camera pair), IR cameras, depth cameras, and combinations thereof.

[0048] In another embodiment, computer 106 acts as a thin client communicating with cloud gaming provider 112 over a network. Cloud gaming provider 112 maintains and executes the video game being played by user 102. Computer 106 transmits input from HMD 102, interface object 104, and camera 108 to the cloud gaming provider, which processes the input to influence the game state of the executing video game. Output from the executing video game, such as video data, audio data, and haptic feedback data, is transmitted to computer 106. Computer 106 may further process the data before transmission, or it may directly transmit the data to the relevant device. For example, video and audio streams may be provided to HMD 102, while vibration feedback commands may be provided to interface object 104.

[0049] In one implementation, HMD 102, interface object 104, and camera 108 may themselves be networked devices connected to network 110 to communicate with cloud gaming provider 112. For example, computer 106 may be a local network device such as a router, which does not perform video game processing but facilitates the passage of network traffic. The connection of HMD 102, interface object 104, and camera 108 to the network may be wired or wireless.

[0050] Additionally, while the embodiments described in this disclosure may be referenced to heads-up displays, it should be understood that in other embodiments, non-heads-up displays may be replaced, including but not limited to televisions, projectors, LCD displays, portable device screens (e.g., tablets, smartphones, laptops, etc.), or any other type of display that may be configured to render video and / or provide a display of interactive scenes or virtual environments according to embodiments of the present invention.

[0051] Figure 2A head-up display (HMD) according to an embodiment of the present disclosure is shown. As shown, HMD 102 includes a plurality of lamps 200A-H. Each of these lamps can be configured to have a specific shape and can be configured to have the same or different colors. Lamps 200A, 200B, 200C, and 200D are arranged on the front surface of HMD 102. Lamps 200E and 200F are arranged on the side surfaces of HMD 102. Lamps 200G and 200H are arranged at the corners of HMD 102 to span the front and side surfaces of HMD 102. It should be understood that the lamps can be identified in captured images of an interactive environment in which the user uses HMD 102. Based on the identification and tracking of the lamps, the position and orientation of HMD 102 in the interactive environment can be determined. It should also be understood that, depending on the specific orientation of HMD 102 relative to the image capture device, some lamps may or may not be visible. Furthermore, depending on the orientation of the HMD102 relative to the image capture device, different parts of the lamps (e.g., lamps 200G and 200H) can be exposed for image capture.

[0052] In one implementation, the lights can be configured to indicate the current state of the HMD to other people nearby. For example, some or all the lights can be configured to have a specific color arrangement, intensity arrangement, be configured to flash, have a specific on / off configuration, or other arrangements indicating the current state of the HMD 102. For instance, the lights can be configured to display different configurations during active game events (typically occurring during the active timeline or within the game scene) compared to other inactive game events, such as navigation menu interfaces or configuring game settings (during which the game timeline or scene may be inactive or paused). The lights can also be configured to indicate the relative intensity level of a game event. For example, the intensity or flashing rate of the lights can increase as the intensity of the game event increases. In this way, people other than the user can observe the lights on the HMD 102 and understand that the user is actively engaged in an intense game event and may not wish to be disturbed at that moment.

[0053] HMD 102 may additionally include one or more microphones. In the illustrated embodiment, HMD 102 includes microphones 204A and 204B defined on the front surface of HMD 102, and microphone 204C defined on the side surface of HMD 102. By utilizing a microphone array, the sound from each microphone can be processed to determine the location of a sound source. This information can be used in various ways, including excluding unwanted sound sources, associating sound sources with visual recognition, etc.

[0054] HMD 102 may also include one or more image capture devices. In the illustrated embodiment, HMD 102 is shown as including image capture devices 202A and 202B. By utilizing the stereoscopic image capture device pair, three-dimensional (3D) images and videos of the environment can be captured from the perspective of HMD 102. Such videos can be presented to the user to provide a “video see-through” capability when the user is wearing HMD 102. That is, although strictly speaking, the user cannot see through HMD 102, the video captured by image capture devices 202A and 202B can still provide an equivalent function of being able to see the environment outside HMD 102 as if seeing through HMD 102. Such videos can be enhanced with virtual elements to provide an augmented reality experience, or can otherwise be combined or blended with virtual elements. Although two cameras are shown on the front surface of HMD 102 in the illustrated embodiment, it should be understood that any number of outward-facing cameras can be mounted on HMD 102 in any direction. For example, in another embodiment, a camera can be mounted on the side of the HMD 102 to provide additional panoramic image capture of the environment.

[0055] In some implementations, an outward-facing camera is used to track the location and / or orientation of the HMD. In other implementations, the HMD uses Simultaneous Localization and Mapping (SLAM) technology or other methods to determine its location / orientation in the local environment.

[0056] In some implementations, the HMD includes one or more inertial sensors to enable the detection and tracking of the HMD's movement.

[0057] In some implementations, the HMD includes a magnetic sensor configured to detect magnetic fields / signals generated by one or more magnetic emitters located in the local environment. By sensing the magnetic field / signal, the location and / or orientation of the HMD can be determined.

[0058] Figure 3 This conceptually illustrates the functionality of an HMD 102 combined with a running video game according to an embodiment of this disclosure. The running video game is defined by a game engine 320, which receives input to update the game state of the video game. The game state of the video game can be defined, at least in part, by various parameter values ​​of the video game that define aspects of the current game plot, such as the presence and location of objects, conditions of the virtual environment, event triggering, user profiles, viewing angle, etc.

[0059] In the illustrated embodiment, the game engine receives controller input 314, audio input 316, and motion input 318 by way of example. Controller input 314 can be defined based on the operation of a game controller separate from the HMD 102, such as a handheld game controller (e.g., a Sony DUALSHOCK®4 wireless controller, a Sony PlayStation® Move motion controller) or interface object 104. For example, controller input 314 may include directional input, button presses, trigger activation, movement, gestures, or other types of input processed from the operation of the game controller. Audio input 316 can be processed from microphone 302 of the HMD 102, or from microphones included in camera 108 or elsewhere in the local environment. Motion input 318 can be processed from motion sensor 300 included in the HMD 102, or from image capture device when camera 108 captures an image of the HMD 102. The game engine 320 receives input processed according to the game engine's configuration to update the game state of the video game. The game engine 320 outputs game state data to various rendering modules, which process the game state data to define the content to be presented to the user.

[0060] In the illustrated embodiment, the video rendering module 322 is defined to render a video stream for presentation on the HMD 102. The video stream may be presented by the display / projector assembly 310 and viewed by the user's eyes 306 through optics 308. The speaker 304 is configured to render an audio stream for the user to listen to. In one embodiment, the audio stream is output through the speaker 304 associated with the HMD 102. It should be understood that the speaker 304 may take the form of an outdoor loudspeaker, headphones, or any other type of speaker capable of delivering audio.

[0061] In one embodiment, the HMD 102 includes a gaze-tracking camera 312 to enable tracking of a user's gaze. The gaze-tracking camera captures images of the user's eyes, and these images are analyzed to determine the user's gaze direction. In one embodiment, information about the user's gaze direction can be used to influence video rendering. For example, if it is determined that the user's eyes are looking in a particular direction, video rendering in that direction can be prioritized or emphasized, for example, by providing more detail or faster updates in the area the user is looking at. It should be understood that the user's gaze direction can be defined relative to the head-up display, relative to the user's real-world environment, and / or relative to a virtual environment rendered on the head-up display.

[0062] Generally, when considered alone, analyzing the images captured by the gaze-tracking camera 312 provides the user's gaze direction relative to the HMD 102. However, when considering the tracking position and orientation of the HMD 102 in conjunction with these factors, the user's real-world gaze direction can be determined, since the position and orientation of the HMD 102 are synonymous with the position and orientation of the user's head. That is, the user's real-world gaze direction can be determined by tracking the positional movement of the user's eyes and tracking the position and orientation of the HMD 102. When rendering a view of the virtual environment on the HMD 102, the user's real-world gaze direction can be applied to determine the user's virtual-world gaze direction within the virtual environment.

[0063] Additionally, the haptic feedback module 326 is configured to provide signals to the HMD 102 or to another user-operated device, such as haptic feedback hardware included in a controller. Haptic feedback can take various forms, such as vibration feedback, temperature feedback, pressure feedback, etc.

[0064] Figure 4 This illustration conceptually depicts asynchronous interaction between users on a head-up display according to embodiments of the present disclosure. In the illustrated embodiment, a first user 400 plays a video game, where the first user 400 controls a vehicle. For example, the video game could be a racing game, where the first user 400 races the speed of vehicle 406 (e.g., a car, boat, airplane, spacecraft, etc.) along route 408-1. By way of example and not limitation, in the illustrated embodiment, vehicle 406 takes the form of a car having a driver's seat and multiple passenger seats.

[0065] A conceptual top view of vehicle 406 is shown at reference numeral 407. A first user 400 controls a virtual character 408 that drives vehicle 406. Virtual character 408 sits in driver's seat 410 of vehicle 406. Therefore, view 412 is from the perspective of virtual character 408 in driver's seat 410, which defines the first view position in the virtual environment of the video game. As an example and not a limitation, when the first user 400 looks forward, the first user 400 can see virtual character 408's hands operating the steering wheel of vehicle 406.

[0066] It should be understood that the view 412 of the first user during the game's progression can be recorded as video. This recorded video can be later played back and viewed by another user. However, when the latter user is another HMD user, such playback cannot utilize the capabilities of the HMD to provide a more engaging experience. Furthermore, the forced movement of the view may cause discomfort or discomfort to the latter user. Therefore, according to embodiments of this disclosure, an enhanced asynchronous experience can be provided to HMD users, surpassing the ability to simply replay the recorded video of the first user 400's game progression.

[0067] For example, in the illustrated embodiment, a second user 420 can watch the game plot of a first user 400, but as a passenger in vehicle 406. The second user 420 uses an HMD 422 to watch a replay of the first user 400's game plot. The second user 420 can also operate a controller device 424 to provide input and commands during the replay. As a passenger in vehicle 406, the second user 420 can be represented by and control a virtual character 426 located in passenger seat 428 within vehicle 406. Therefore, a view 430 of the virtual environment is defined from the perspective of a second viewpoint in vehicle 406, namely the position of the virtual character 426 in passenger seat 428.

[0068] It should be understood that the view 430 provided to the second user 420 is a novel perspective that did not exist prior to the initial game scenario of the first user 400 and was not generated during the initial game scenario of the first user 400. Furthermore, the view 430 can respond to the tracked movement of the HMD 422, allowing the second user 420 to look around and in different directions during replay. In some embodiments, the position and / or orientation of the HMD 422 is tracked and used to determine the view position and / or orientation of the view 430. It should be understood that because the view 430 is viewed from the perspective of the passenger virtual character 426 in the vehicle 406, the view position of the second user 420's view 430 is associated with the position and movement of the vehicle 406 in the virtual environment of the video game. In some embodiments, the view position of the view 430 is limited by the boundaries of the vehicle 406. For example, there may be limited space within vehicle 406, and the view position of view 430 can be moved within said limited space, so that, for example, in response to the second user 420 moving HMD 422, the view position cannot be moved outside vehicle 406.

[0069] In some implementations, the second user 420 may provide comments during replay. Furthermore, these comments may be associated with a specific replay time during which the comment is made, so that other users watching the replay (which may include the first user 400) can see the comments at the time they were made while watching the replay.

[0070] By enabling the second user 420 to view the game scenario of the first user 400 from a different perspective (the perspective of a passenger in the same vehicle), the second user 420 can experience a greater sense of participation and interaction in the game scenario, even though the interaction is asynchronous. This is because it is natural to be a passenger in a vehicle driven by another person, whereas it would be unnatural to occupy the driver's perspective when the person is not actually the driver.

[0071] In some implementations, additional passenger seats, such as passenger seats 432a, 432b, and 432c, may be present in vehicle 406. These passenger seats may be occupied by additional virtual avatars representing additional users 440. It should be understood that such additional users 440 will be able to view the replay from the perspective defined by their respective virtual avatars in their seats in vehicle 406. In some implementations, when multiple users simultaneously occupy vehicle 406, they can interact with each other, for example, through conversation and / or text chat. Each user is represented by their respective virtual avatar or surrogate, which each user can customize according to their preferences. Therefore, users simultaneously acting as passengers in vehicle 406 will be able to see each other's avatars and interact with each other as if they were passengers in a real vehicle. In this way, a shared experience asynchronous to the initial game scenario played by the first user 400 can be provided. For example, by using controller devices, user motion tracking, etc., users can further control the gestures of their corresponding virtual avatars / surrogates through various control mechanisms.

[0072] In some embodiments, the view of the second user 420 can be recorded when the second user 420 (and / or other additional users as described above) views the game sequence of the first user 400 as a passenger in vehicle 406. According to embodiments of this disclosure, other users can access and view this recorded ride video.

[0073] In some implementations, when a second user 420 (and / or other additional users as described above) views the game storyline of a first user 400 who is a passenger in vehicle 406, the interactions of the second user 420 during viewing can be recorded. For example, the movement of the second user 420's virtual character / avatar 426, audio spoken by the second user 420, comments made by the second user 420, etc., can be recorded. Using this information, along with the recorded game storyline of the first user 400, another user can then asynchronously view the game storyline of the first user 400 and the interactions of the second user 420 while the second user 420 is viewing the game storyline of the first user 400. In this way, subsequent asynchronous interactions of the second user 420 can be obtained in a manner similar to how the first user's game storyline is presented to the second user, for example, for one or more additional passengers (users 440) in seats 432a, 432b, and 432c. Thus, the additional passengers can view an aggregated replay of the first user 400 and the second user 420.

[0074] As an example, and not a limitation, the ability to view aggregated asynchronous interactions is useful for providing game tutorials where a player's gameplay is commented on by another player. For example, as described above, a player's gameplay is recorded as they traverse a track, including metadata and game state (e.g., vehicle position, player's gestures, etc.). Then, at a later point, a second user 420, acting as a passenger in a car, uses the captured metadata to comment on the first user 400's driving (e.g., instructing the first user 400 to miss a peak, not to swerve wide enough at a turn leading to the highway, to accelerate at a certain point, to make a sharp turn at a specific location, etc.), and such audio and gestures can be captured. Then, at a later point in time, one or more passengers in the back seats 432a / b / c, watching the first user 400's driving and the second user 420's interactions, can learn from the driving and the comments / instructions. For example, passengers can see the movements, gestures, and any other recorded interactions of the corresponding avatars of the first and second users.

[0075] Therefore, one application of viewable aggregated asynchronous interactions is enabling users to comment on each other's game scenarios so that other users can learn from the game scenarios and comments. While the foregoing example has been described with reference to driving simulations, it should be understood that these concepts can be applied to any type of game or interactive application that can record asynchronous game scenarios / interactions and make them available for later viewing.

[0076] Figure 5 This illustration conceptually depicts a change in the view orientation of a first user during a game scenario session according to an embodiment of this disclosure, and a change in the view orientation of a second user when watching a replay of the first user's game scenario session. In the illustrated embodiment, a path 500 of the first user 400 through the virtual environment is shown. At various points in time, the view orientation of the first user 400 is indicated by corresponding arrows. At time t0, the first user 400 (or representing the first user 400 or a virtual character / avatar controlled by the first user 400) is shown in the virtual environment with position P0 and view orientation D0. At time t1, the first user 400 has moved to position P1 and has view orientation D1. At time t2, the first user 400 has moved to position P2 and has view orientation D2. At time t3, the first user 400 has moved to position P3 and has view orientation D3.

[0077] When using an HMD to watch a replay of another user's game, simply watching a recording of the user's game can cause discomfort or unease for the viewer because this first-person perspective of the game can change too quickly or too abruptly in terms of view position and orientation, causing discomfort. Therefore, in some implementations, a replay is provided in which the orientation of the view is controlled by the viewer.

[0078] For example, in some embodiments, the view of the second user 420 during the replay of the game scene by the first user 400 is from the same position at a given time during the game scene, but the view orientation (or view orientation) of the second user 420 can be determined by the second user 420. Therefore, referring to... Figure 5 When the second user 420 is watching a replay of the game scene played by the first user 400, the second user 420's view position follows the same path 500 through the virtual environment from position P0 to P3 and further, as shown in the figure. However, during the replay, the second user 420's view orientation / orientation does not necessarily follow the same view orientation as the first user 400 (i.e., from D0 to D3 and further).

[0079] In some implementations, the view orientation of the second user 420 is completely separated from that of the first user 400, such that the view orientation during playback is entirely controlled by the second user 420. It should be understood that the view orientation of the second user 420 can be controlled in response to various types of inputs, such as tracking movement of the second user 420's HMD device, inputs from a controller device operated by the second user 420, etc.

[0080] In some implementations, the view orientation of the second user 420 is controlled partly by the view orientation of the first user 400 and partly by the second user 420. For example, in some implementations, the view orientation of the second user 420 is generated by a combination of the view orientation of the first user 400 and input from the second user 420. In some implementations, when the second user 420's HMD is in a predefined initial orientation (e.g., homepage orientation or when the second user 420 is looking forward), the second user 420's view orientation is the same as the first user 400's view orientation, but deviates from the first user 400's view orientation based on a change in the orientation of the second user 420's HMD. For example, when the second user 420 moves their HMD by turning their head to look right, the second user 420's view orientation turns to the right relative to the first user 400's view orientation during a playback of the first user 400's game scene.

[0081] In some implementations, the orientation change of the first user 400's view during playback is limited to a maximum rate of change. For example, in some implementations, during playback viewing by the second user 420, the first user 400's view orientation change is allowed not to exceed a predefined maximum rate of change, while changes exceeding the predefined maximum rate of change are reduced to the predefined maximum rate of change. In these implementations, the second user 420's view orientation may lag behind the first user 400's view orientation for a specific time period, and if the first user 400's view orientation changes to a new orientation during such a time period, the second user 420's view orientation is adjusted accordingly according to the same predefined maximum rate to move toward the new orientation.

[0082] In some implementations, the view orientation of the second user 420 during playback can be configured to switch between various selectable modes in response to user input from the second user 420, including, by way of example and not limitation: being locked to the view orientation of the first user 400, being completely separated from the view orientation of the first user 400 and completely controlled by the second user 420, and / or being partially controlled by the view orientation of the first user 400 and partially controlled by the second user 420. In some implementations, the view orientation of the second user 420 can be switched in response to button input from a controller device operated by the second user 420.

[0083] In some implementations, during the replay of a game scene, the second user 420's view orientation is the same as the first user 400's view orientation until the second user 420 moves its HMD (e.g., beyond a threshold amount away from the initial orientation), at which point the second user 420's view orientation becomes separate from the first user 400's view orientation (and is completely controlled by the second user 420). To return to the first user 400's view orientation, the second user 420 can provide predefined inputs, without limitation, such as pressing a button, performing a predefined gesture, or uttering a verbal command.

[0084] Continue to refer to Figure 5 Graph 502 shows the change in the horizontal angle (in degrees) of the view direction of the first user 400 and the second user 420 over time, such that when the second user 420 watches a replay of the game scene of the first user 400, the view direction of the second user 420 is independent of the view direction of the first user 400. The angle of the view direction of the first user 400 is shown by curve 504, and the angle of the view direction of the second user 420 is shown by curve 506.

[0085] At time t0, the first user 400 has a view orientation D0 at a 0-degree angle, and the second user 420 also has the same view orientation at a 0-degree angle. However, as time progresses from t0 to t1, the view orientation of the first user 400 changes to D1 at an angle of approximately 90 degrees; while the view orientation of the second user 420, independent of the view orientation of the first user 400, remains unchanged at 0 degrees. At time t2, the view orientation D2 of the first user 400 has moved to approximately -60 degrees, while the view orientation of the second user 420 has moved to approximately 150 degrees. At time t3, the view orientation D3 of the first user 400 has moved to approximately -15 degrees, while the view orientation of the second user 420 has moved to approximately 90 degrees.

[0086] As an example and not a limitation, at time t0, the second user 420 is shown with view 508. As described above, this view is the same as the view of the first user 400. In some embodiments, an indicator 510 is provided to indicate the direction of the first user 400's view relative to the view direction of the second user 420. In the illustrated embodiment, the indicator 510 is in the form of a bar, with a pointer indicating degrees, which shows the direction of the first user 400's view at that time during playback. As shown, the pointer is at zero because the second user 420's view direction is the same as the first user 400's view direction.

[0087] At time t1, the second user 420 is shown with view 512. At this time, as previously discussed, the view orientation of the second user 420 is 0 degrees, while the view orientation of the first user 400 is 90 degrees. Therefore, the view orientation of the first user 400 is now +90 degrees relative to the view orientation of the second user 420. In the illustrated embodiment, the indicator 510 thus displays a pointer of 90 degrees, indicating the horizontal angle of the view orientation of the first user 400 relative to the view orientation of the second user 420 at time t1. Alternatively, in some embodiments, an additional indicator may be provided, for example, in the form of a visual cue such as arrow 514, indicating the direction of the view orientation of the first user 400 relative to the view orientation of the second user 420.

[0088] Although embodiments have been described with respect to horizontal view orientation, it should be understood that the principles of this disclosure can also be applied to vertical view orientation. For example, indicator 510 can also indicate the vertical view orientation of the first user 400.

[0089] Figure 6AA conceptual top view of a virtual character in a virtual environment according to an embodiment of the present disclosure is shown, illustrating the relationship between view position and view orientation in the virtual environment. In the illustrated embodiment, virtual character 408 represents a first user 400, and the view position or perspective and view orientation of the first user 400's view are also defined. Initially, the first user 400's view has a view orientation E1 in the virtual environment.

[0090] In some embodiments, a second user 420, who is watching the game plot of the first user 400, may be represented by a virtual character 426. The virtual character 426 shown defines the view position or perspective and view orientation of the second user 420's view. In the illustrated embodiment, at the initial time, the second user 420's view has a view orientation F1 in the virtual environment that is substantially similar to the view orientation E1.

[0091] In some embodiments, the view of the second user 420 is configured to have a predefined spatial relationship with the view of the first user 400. This predefined spatial relationship can be defined by certain predefined parameters, such as a predefined distance between the view position of the second user 420 and the view position of the first user 400, and a predetermined angular position of the view position of the second user 420 relative to the view position of the first user 400. As shown in the illustrated embodiment, the corresponding virtual characters of the first user 400 and the second user 420 define their view positions, and virtual character 426 can have a predefined spatial relationship with virtual character 408. As shown, virtual character 408 has a position G1 in the virtual environment, and virtual character 426 is separated from virtual character 408 by a distance L in the virtual environment at position G2. Furthermore, virtual character 426 is laterally located to the left of virtual character 408.

[0092] The virtual character 408 in the view of the first user 400 is shown facing direction E1. If direction E1 is considered to be zero degrees, then while the game plot of the first user 400 is being replayed, the position of the virtual character 426 in the view of the second user 420 has an angular position of approximately -90 degrees relative to the virtual character 408 (i.e., approximately 90 degrees counterclockwise when considered from a top-down perspective). By positioning the virtual character 426 near the virtual character 408 at a predefined relative position, the second user 420 can experience the replay in a more natural and engaging way compared to if the second user 420 were viewing the replay from the specific perspective of the virtual character 408. Therefore, an enhanced experience can be provided even though the interaction is asynchronous.

[0093] However, such an arrangement raises the question of how to maintain this spatial relationship throughout the game's plot. Maintaining this spatial relationship firmly in real-time would mean that the view position of the second user 420, defined by the virtual character 426, could move very rapidly due to the rapid movement (change in position and / or orientation) of the virtual character 408 in order to maintain the spatial relationship in real-time. This could cause the second user 420 to feel uncomfortable due to excessive movement of their view position. For example, if the first user 400 rapidly rotates the virtual character 408 90 degrees clockwise to the view direction E2 (e.g., by operating the controller or rotating the HMD), and maintains the spatial relationship between the virtual character 426 and the virtual character 408 in real-time, then the virtual character 426 will simultaneously move from position G2 (and view direction F1) to position G3 (and view direction F2), thus rapidly traversing path 600 in the process. In this case, it can be seen that even a relatively small change in the orientation of the virtual character 408 can result in significant movement of the virtual character 426.

[0094] Therefore, in some embodiments, the view position and / or orientation of the second user 420 is allowed to drift from its predefined spatial relationship with the view position and / or orientation of the first user 400. For example, in some embodiments, the virtual character 426 moves in response to the movement of the virtual character 408 to maintain the predefined spatial relationship, but the movement of the virtual character 426 is limited to a predefined maximum movement speed in the virtual environment. In some embodiments, the movement of the virtual character 426 is limited to a predefined maximum acceleration (change in velocity). It should be understood that velocity / acceleration can involve both translation and angular movement of the virtual character 426 in the virtual environment.

[0095] When the spatial relationship between virtual character 426 and virtual character 408 is not maintained in real time (e.g., by setting the maximum speed / acceleration as described above), the position / orientation of virtual character 426 will lag behind its expected (or correct) position / orientation based on the predefined spatial relationship. For example, as indicated in the illustrated embodiment, when virtual character 408 rotates 90 degrees clockwise, the expected position / orientation of virtual character 426 is at position G3 and direction F2. However, since the spatial relationship cannot be maintained in real time, there may be a situation where virtual character 408 has rotated 90 degrees clockwise while virtual character 426 has not yet reached position G3 and direction F2.

[0096] Therefore, the virtual character 426 can traverse the more direct path 602 faster than path 600 to reach position G3 and orientation F2. In some implementations, for each frame (or a predetermined number of frames or a predetermined amount / unit of elapsed time), an optimal direction (e.g., the direction requiring the least time to reach the desired position / orientation) from the virtual character 426's current position / orientation is determined, and the virtual character 426 traverses the virtual environment in the optimal direction (e.g., constrained by, for example, maximum speed / acceleration). For example, the position / orientation of the virtual character in the next frame can be linearly interpolated along the optimal direction. It should be understood that the optimal direction of movement of the virtual character 426 can continue to change from one frame to the next as the virtual character 408 can continue to move. In this way, the virtual character 426 moves towards its desired position / orientation in an efficient manner to maintain predetermined spatial relationships.

[0097] Figure 6B The graphs shown illustrate the horizontal angular orientation of virtual character 408 (reference 604) and the angular position of virtual character 426 relative to the angular orientation of virtual character 408 according to embodiments of the present disclosure (reference 606). Again, as already noted, the orientations of virtual characters 408 and 426 are synonymous with the view directions of the first user 400 and the second user 420, respectively. At an initial time t0, virtual character 408 exhibits an orientation of zero degrees (direction E1), and virtual character 426 exhibits an angular position of -90 degrees relative to the position of virtual character 408 (based on a predefined spatial relationship between virtual character 426 and virtual character 408). From time t1 to t2, the angular orientation of virtual character 408 changes from zero degrees to 90 degrees (rotating from E1 to E2). During the time interval from t1 to t2, the angular position of virtual character 426 relative to virtual character 408 may lag behind the expected angular position (which, according to the predefined spatial relationship, would be -90 degrees), thus increasing to -165 degrees in the negative direction before tending back to the predefined spatial relationship of -90 degrees.

[0098] Figure 7A conceptual illustration of an embodiment of this disclosure shows that a second user 420 can navigate and watch independently of the first user while viewing a replay of a first user's game sequence in a virtual environment. At the initial time of the replay, the first user 400 has a view from position P0 in the virtual environment. As noted, the first user 400's view may be defined by the position and orientation of a representative virtual character 408. During the replay of the first user 400's game sequence, in some embodiments, the second user 420 is able to navigate its view independently of the first user 400's view. This may include navigating the position and orientation of the view independently of the first user 400's view. The second user 420's view may be defined by a representative virtual character 426, which is navigated independently by the second user 420 in the virtual environment. Because the second user 420 can independently determine its view, for example by moving the virtual character 426, unlike the previously described embodiments, the position and orientation of the second user 420's virtual character 426 may not follow the position and orientation of the first user 400's virtual character 408, and there is no maintained spatial relationship between the virtual characters.

[0099] Therefore, for example, when virtual character 408 moves from position P0 to position P1, virtual character 426 may not follow. Furthermore, the second user 420's view of the virtual environment may become more detached from the first user 400's view because the position of virtual character 426 becomes further away from the position of virtual character 408. Therefore, in some implementations, an indicator is provided in the virtual environment to prompt the second user 420 to move toward the position of the first user 400 (e.g., to move virtual character 426 toward the position of virtual character 408).

[0100] In some implementations, a graphical notification is displayed to the second user 420, indicating that they are viewing or moving away from the location of the virtual character 408 of the first user 400, and / or prompting the second user 420 to move towards the virtual character 408. In some implementations, this graphical notification 700 is rendered in or near the location currently being viewed by the second user 420 within the virtual environment. In some implementations, an audio notification may be provided to prompt the second user 420 to move towards the first user 400.

[0101] In some implementations, such a notification is triggered when the distance between virtual character 426 and virtual character 408 exceeds a predefined threshold distance in the virtual environment. For example, when virtual character 408 is at position P0 and virtual character 426 is at position P2, the distance between the virtual characters d1 may be less than the threshold distance. Therefore, even if the second user 420 is not looking in the same direction as or towards the first user 400, no notification is provided because virtual character 426 is still close to virtual character 408. However, when virtual character 408 moves to position P1, the distance between virtual character 426 and virtual character 408 increases to exceed the predefined threshold distance d2. Therefore, a notification is provided to the second user 420 to prompt or otherwise encourage the second user 420 to move towards the first user 400.

[0102] In some implementations, other forms of notification or prompting may be provided to indicate that the second user 420 is moving toward the first user 400. For example, the rendering of the virtual environment may be adjusted to make the area 702 that the second user 420 is viewing less visually appealing, thereby encouraging the second user 420 to look elsewhere and move toward the first user 400. Various rendering properties / parameters may be adjusted to make the scene less visually appealing, such as reducing color saturation, reducing contrast, reducing lighting levels, simplifying surface textures, reducing the number / density of virtual objects, reducing graphics complexity, etc. In some implementations, audio may be adjusted to reduce auditory interest, for example by reducing the volume level of the sound from area 702, adjusting audio settings (e.g., adjusting the frequency, such as reducing treble / high frequencies to make the sound softer), reducing the volume or eliminating background audio tracks (e.g., background music), reducing the number of sounds generated, etc.

[0103] In some implementations, the rendering of the virtual environment is adjusted based on the position of the virtual character 426 relative to the position of the virtual character 408. For example, graphics 704 conceptually represent the virtual environment and are centered on the position of the virtual character 426 within the virtual environment. In some implementations, the rendering of the virtual environment is adjusted to reduce visual interest beyond a specific boundary 706, the distance of which from the virtual character 426 varies based on the relative positions of the virtual character 426 and the virtual character 408. In some implementations, the boundary 706 is farther from the virtual character 426 when the direction (radial direction from the virtual character 426 to the boundary position) is toward the virtual character 408, and the distance decreases when the direction is away from the virtual character 408.

[0104] In some implementations, the rendering of the virtual environment is adjusted based on the position and / or orientation of the virtual character 408 (e.g., as noted above, to reduce the level of visual interest). For example, in some implementations, regions in the virtual environment beyond a predetermined distance from the position of the virtual character 408 are rendered in an adjusted manner to reduce visual interest in these regions. In some implementations, a region of interest 708 may be defined based on the position and / or orientation of the virtual character 408. The region of interest is rendered with normal settings, while regions falling outside the region of interest 708 are rendered with adjusted settings, thereby reducing visual interest in these regions.

[0105] In some implementations, the region of interest is defined based on the position and orientation of the virtual character 408. For example, the distance from the position of the virtual character 408 defining the boundary of the region of interest increases in the orientation direction of the virtual character 408 and decreases in the direction opposite to the orientation of the virtual character 408.

[0106] Figure 8 This illustration conceptually depicts a virtual character in a virtual environment during playback of a game scenario for a first user 400, with rendering adjustments made according to an embodiment of this disclosure. In the illustrated embodiment, rendering is adjusted to reduce visual interest, and the adjustment amount increases with distance from the virtual character 426. Furthermore, in the illustrated embodiment, several concentric curves 800, 802, 804, 806, and 808 are equidistant lines representing the adjustment amount. Along a given curve, the rendering adjustment amount is the same, where the outer curve indicates a larger adjustment level than the inner curve (e.g., the rendering adjustment amount along curve 808 is greater than that along curve 806, the rendering adjustment amount along curve 806 is greater than that along curve 804, etc.). In some embodiments, the concentric curves may indicate audio adjustment amounts for reducing auditory interest.

[0107] Further, as shown in the figure, in the direction from virtual character 426 toward virtual character 408 (refer to 812), the equidistant curves are further spaced apart. This means that the rendering quality decreases less rapidly with distance in the direction toward virtual character 408 compared to the direction away from virtual character 408 (refer to 814). In some implementations, for a given equidistant curve, its distance from virtual character 426 increases as the direction defined from virtual character 426 to a given point on the equidistant curve becomes more pointed toward virtual character 408.

[0108] Figure 9This illustration conceptually depicts a virtual character in a virtual environment during playback of a game scenario for a first user 400, with rendering adjustments made according to an embodiment of the present disclosure. In the illustrated embodiment, rendering is adjusted to reduce visual interest, and the adjustment amount increases with distance from the virtual character 408. Furthermore, in the illustrated embodiment, several concentric curves 900, 902, 904, 906, and 908 are equidistant lines representing the adjustment amount. Along a given curve, the rendering adjustment amount is the same, where the outer curve indicates a larger adjustment level than the inner curve (e.g., the rendering adjustment amount along curve 908 is greater than that along curve 906, the rendering adjustment amount along curve 906 is greater than that along curve 904, etc.). In some embodiments, the concentric curves may indicate audio adjustment amounts for reducing auditory interest.

[0109] Figure 10 The diagram conceptually illustrates the path traversed by a virtual character of a first user 400 according to an embodiment of the present disclosure, and the path traversed by a virtual character of a second user 420 kept near the virtual character of the first user 400. As shown, virtual character 408 represents the first user 400 in the virtual environment, and virtual character 408 traverses path 1000 of the virtual environment during a game scenario session.

[0110] In some implementations, when a second user 420 is watching a replay of a game scene played by a first user 400, the area 1002 of the virtual environment closest to the virtual character 408 can be prioritized for viewing. For example, the area 1002 can be rendered using normal rendering settings, while areas outside the area 1002 can be rendered using adjusted settings to reduce visual interest.

[0111] During the process of traversing path 1000 in the virtual environment, the virtual character 408 can move from position 1004 to position 1006 and then to position 1008, as shown in the illustrated embodiment. In some embodiments, the preferred area 1002 of the virtual environment is defined substantially by the area of ​​the virtual environment that the first user 400 views (or that the virtual character 408 faces) during the first user 400 session.

[0112] As already noted, in some embodiments, the view of the second user 420 may be the same as the view of the first user 400 during the session of the first user 400. However, in some embodiments, the second user 420 may change its view to be independent of the view of the first user 400. For example, when the virtual character 408 moves from position 1004 to position 1006, the second user 420 may move its view position to position 1016, thereby deviating from the view of the first user 400. This can be defined by the second user 420's view position moving from the virtual character 408 to position 1016. In doing so, the second user 420 can be represented by the virtual character 426 in the virtual environment, and therefore the view position of the second user 420 is defined by the position of the virtual character 426, and the view orientation of the second user 420 is defined by the orientation of the virtual character 426.

[0113] In some implementations, as the virtual character 408 moves along path 1000 through the virtual environment, a window or region is defined around the virtual character 408, and the virtual character 426 of the second user 420 is confined within said window or region. That is, when the virtual character 408 moves, the virtual character 426 needs to be in some proximity to the virtual character 408. As an example, and not a limitation, in the illustrated implementation, when the virtual character 408 is at position 1004, the virtual character 426 is confined to region 1010, which is close to the virtual character 408. Then, when the virtual character 408 moves to position 1006, the region shifts to region 1012, which is close to the virtual character 408, and the virtual character 426 is confined within said region. When the virtual character 408 moves to position 1008 and the proximity region shifts to region 1014, the virtual character 426 can be forced to move from position 1016 to position 1018 to remain within the vicinity of the virtual character 408.

[0114] Figure 11This illustration conceptually depicts virtual characters in a virtual environment according to embodiments of the present disclosure, and variations in the areas in which virtual characters can be located. In the illustrated embodiment, the positions of virtual characters 408 and 426 define the view positions of the first user 400 and the second user 420, respectively. Virtual character 408 is then located at position 1100, thus allowing virtual character 426 to move within region 1108, which includes (and may be close to) virtual character 408. In some embodiments, the outer boundary of region 1108 is defined by a predefined distance from the position of virtual character 408. For example, virtual character 426 can move from position 1104 to another position 1106 within the virtual environment, since both positions are within region 1108. However, when virtual character 408 is located at position 1102, virtual character 426 is not allowed to move beyond the boundary of region 1108, and therefore the view position of the second user 420 is confined therein.

[0115] Subsequently, virtual character 408 moves away from virtual character 426 from position 1100 to position 1102, while virtual character 426 remains at position 1106. Instead of shifting region 1108 and forcing virtual character 426 to remain within the shifted region during movement, the region is expanded to include both virtual character 408 and virtual character 426 in region 1110, such that the boundary reaches the position of virtual character 426 but does not expand further (away from virtual character 408). Therefore, virtual character 426 is not forced to move in response to the movement of virtual character 408. However, virtual character 426 is not allowed to move beyond the boundary of the expanded region 1110, and therefore virtual character 426 is not allowed to move further away from virtual character 408.

[0116] When virtual character 426 moves from position 1106 to position 1107 toward virtual character 408, the boundary of the shifted area is adjusted to define an area 1112 within which virtual character 426 is allowed to move. That is, the area shrinks as virtual character 426 moves toward virtual character 408. In some embodiments, the area within which virtual character 426 is allowed to move is shrunk as virtual character 426 moves toward virtual character 408 until virtual character 426 is within a predetermined distance of virtual character 408. In other words, the boundary of the allowed area is shifted such that its outer extent is defined by the position of virtual character 426 until virtual character 426 is within a predefined area close to virtual character 408, at which point the boundary is defined by parameters of the predefined area (e.g., a predefined distance from virtual character 408). In this way, the area within which virtual character 426 is allowed to move has a “normal” setting (e.g., a predefined distance from virtual character 408 or other area shape with respect to the orientation of virtual character 408), which is maintained while virtual character 426 is within this area. However, depending on whether virtual character 408 or virtual character 426 is starting to move, the area can be expanded or shrunk as needed. If virtual character 408 begins to move away from virtual character 426, the area can be expanded, and if virtual character 426 begins to move towards virtual character 408, the area can be shrunk.

[0117] Figure 12 A system for enabling asynchronous interaction between HMD users according to an embodiment of the present disclosure is shown. As shown, client device 1200 executes a first session 1202 of a video game. Game plot metadata generated from the execution of the first session 1202 of the video game is recorded. The game plot metadata may include game state values ​​generated by the execution of the first session of the video game.

[0118] The execution of the first session is driven by a first user 1208 using a first head-up display (HMD) 1204 through an interactive game plot of a video game. The execution of the first session renders a first view of the virtual environment of the video game for presentation via the first HMD 1204. The first view is defined from a first position in the virtual environment determined by the interactive game plot, and the first view is also based on the tracking movement of the first HMD 1204. The tracking movement of the first HMD 1204 may include the tracking orientation of the first HMD 1204 in a first local environment in which the first HMD 1204 is positioned, such that the orientation of the first view in the virtual environment is determined by the tracking orientation of the first HMD 1204.

[0119] The execution of the first session 1202 may include processing input data generated by the first user 1208 from the interactive game scenario, and the game scenario metadata may include such input data. The input data may be generated via an input device, such as a controller 1206 operated by the first user 1208.

[0120] After completing the first session 1202 of the video game, the game plot metadata is transmitted to another client device 1216. Client device 1216 uses the game plot metadata to execute a second session 1218 of the video game to reconstruct game plot events from the first session in the second session. The execution of the second session 1218 renders a second view of the virtual environment for presentation via a second HMD 1220. The second view is defined by a second position in the virtual environment, determined based on a first position in the virtual environment (from the first session). The second view is also defined based on the tracking movement of the second HMD 1220. The tracking movement of the second HMD may include the tracking orientation of the second HMD in a second local environment in which the second HMD is placed, such that the orientation of the second view in the virtual environment is determined by the tracking orientation of the second HMD.

[0121] As described above, in some embodiments, the first position in the virtual environment is a predefined first position in a virtual vehicle placed in the virtual environment, and the second position in the virtual environment is a predefined second position in the virtual vehicle. For example, the predefined first position in the virtual vehicle may be the driver's position in the virtual vehicle, and the predefined second position in the virtual vehicle may be the passenger's position in the virtual vehicle.

[0122] Furthermore, the execution of the second session 1218 may include processing input data generated interactively by the second user 1224. Such input data may be generated via, for example, an input device of the controller 1224 operated by the second user 1224.

[0123] In the illustrated embodiment, the first session 1202 is executed by client device 1200, which is remote from client device 1216 executing the second session 1218. Both client device 1200 and client device 1216 are connected to network 1210. Game plot metadata is transmitted via network 1210. In some embodiments, the game plot metadata generated from the first session 1202 is transmitted via network 1210 to game server 1212, which stores the game plot metadata in cloud storage device 1214. When client device 1216 requests to execute the second session 1218, game server 121 retrieves the game plot metadata from cloud storage device 1214 and transmits the game plot metadata to client device 1216.

[0124] In some implementations, game plot metadata is transmitted directly from client device 1200 to client device 1216 via network 1210.

[0125] In some implementations, the second view of the virtual environment presented by the second HMD 1220 is not necessarily based on the position of the first view; however, the second view is defined by a second position in the virtual environment, which is determined based on the tracked movement of the second HMD 1220. The second position can also be determined using input data generated from the interactivity of the second user 1224 with the second session 1218 of the video game. For example, the input data can be generated by a controller / input device 1222 operated by the second user 1224.

[0126] In some implementations, the rendering of the second view is configured to have settings adjusted based on the orientation of the second view relative to a first position of the first view in the virtual environment. For example, this could include adjusting the level of detail of the second view's rendering such that the level of detail increases when the second view is oriented toward the first position of the first view, and decreases when the second view is oriented away from the first position of the first view. The level of detail can be defined by aspects such as the amount of virtual objects, the amount of color saturation, the amount of texture, the amount of shadows, the resolution level, and / or the graphics complexity.

[0127] refer to Figure 13 This diagram illustrates the components of a head-up display 102 according to an embodiment of the present disclosure. The head-up display 102 includes a processor 1300 for executing program instructions. A memory 1302 is provided for storage purposes and may include both volatile and non-volatile memory. A display 1304 is included, providing a visual interface for a user to view. A battery 1306 is provided as a power source for the head-up display 102. A motion detection module 1308 may include any of various motion-sensitive hardware, such as a magnetometer 1310, an accelerometer 1312, and a gyroscope 1314.

[0128] An accelerometer is a device used to measure acceleration and the reaction force induced by gravity. Single-axis and multi-axis models can be used to detect the magnitude and direction of acceleration in different directions. Accelerometers are used to sense tilt, vibration, and shock. In one embodiment, three accelerometers 1312 are used to provide the direction of gravity, which gives an absolute reference to two angles (world space pitch angle and world space roll angle).

[0129] Magnetometers measure the strength and direction of the magnetic field near the head-up display (HUD). In one embodiment, three magnetometers 1310 are used in the HUD to ensure an absolute reference for the world's yaw angle. In one embodiment, the magnetometers are designed to span the Earth's magnetic field of ±80 microtesla. Magnetometers are affected by metals and provide a monotonic yaw angle measurement via the actual yaw angle. The magnetic field may be distorted due to metals in the environment, which can cause distortion in the yaw angle measurement. If necessary, this distortion can be calibrated using information from other sensors, such as gyroscopes or cameras. In one embodiment, an accelerometer 1312, together with the magnetometers 1310, is used to obtain the tilt and azimuth angles of the HUD 102.

[0130] In some implementations, the magnetometer of the head-up display is configured to be read during periods when electromagnets in other nearby devices are inactive.

[0131] A gyroscope is a device used to measure or maintain orientation based on the principle of angular momentum. In one embodiment, three gyroscopes 1314 provide information about movement across corresponding axes (x, y, and z) based on inertial sensing. Gyroscopes help detect rapid rotation. However, gyroscopes can drift over time without an absolute reference. This requires periodically resetting the gyroscopes, which can be done using other available information, such as position / orientation determination based on object-based visual tracking, accelerometers, magnetometers, etc.

[0132] A camera 1316 is provided for capturing images and image streams of the real-world environment. The head-up display 102 may include more than one camera, including a rear-facing camera (pointed away from the user when the user is viewing the display of the head-up display 102) and a front-facing camera (pointed at the user when the user is viewing the display of the head-up display 102). Additionally, the head-up display 102 may include a depth camera 1318 for sensing depth information of objects in the real-world environment.

[0133] The head-up display 102 includes a speaker 1320 for providing audio output. It may also include a microphone 1322 for capturing audio from the real environment, including sounds from the surrounding environment, user speech, etc. The head-up display 102 includes a haptic feedback module 1324 for providing haptic feedback to the user. In one embodiment, the haptic feedback module 1324 is capable of causing movement and / or vibration of the head-up display 102 to provide haptic feedback to the user.

[0134] An LED 1326 is provided as a visual indicator of the status of the head-up display 102. For example, the LED can indicate battery level, power on, etc. A card reader 1328 is provided to enable the head-up display 102 to read information from and write information to a memory card. A USB interface 1330 is included as an example of an interface for connecting peripheral devices, or connecting to other devices such as other portable devices, computers, etc. In various embodiments of the head-up display 102, any of a variety of types of interfaces may be included to achieve greater connectivity for the head-up display 102.

[0135] The device includes a WiFi module 1332 for connecting to the Internet or a local area network via wireless networking technology. The head-up display 102 also includes a Bluetooth module 1334 for wireless connectivity with other devices. A communication link 1336 may also be included for connecting to other devices. In one embodiment, the communication link 1336 uses infrared transmission for wireless communication. In other embodiments, the communication link 1336 can communicate with other devices using any of a variety of wireless or wired transmission protocols.

[0136] The system includes an input button / sensor 1338 to provide an input interface for the user. This can include any of a variety of input interfaces, such as buttons, touchpads, joysticks, trackballs, etc. An ultrasonic communication module 1340 can be included in the head-up display 102 to facilitate communication with other devices via ultrasonic technology.

[0137] The device includes a biosensor 1342 to detect physiological data from a user. In one embodiment, the biosensor 1342 includes one or more dry electrodes for detecting the user's bioelectrical signals through the user's skin.

[0138] Video input 1344 is configured to receive video signals from the host processing computer (e.g., the main game console) for rendering on the HMD. In some implementations, the video input is an HDMI input.

[0139] The aforementioned components of the head-up display 102 have been described only as exemplary components that may be included in the head-up display 102. In various embodiments of this disclosure, the head-up display 102 may or may not include some of the various components described above. For purposes of convenience of the aspects of this disclosure described herein, embodiments of the head-up display 102 may additionally include other components not currently described but known in the art.

[0140] Figure 14This is a block diagram of a game system 1400 according to various embodiments of the present disclosure. Game system 1400 is configured to provide video streams to one or more clients 1410 via network 1415. Game system 1400 typically includes a video server system 1420 and optionally a game server 1425. Video server system 1420 is configured to provide the video stream to one or more clients 1410 with minimum quality of service. For example, video server system 1420 may receive game commands that change the state or viewpoint within a video game and provide the client 1410 with an updated video stream reflecting this state change with minimal latency. Video server system 1420 may be configured to provide the video stream in various alternative video formats, including formats not yet defined. Furthermore, the video stream may include video frames configured to be presented to the user at various frame rates. Typical frame rates are 30 frames per second, 60 frames per second, and 120 frames per second. However, higher or lower frame rates are included in alternative embodiments of the present disclosure.

[0141] Client 1410, individually referred to herein as 1410A, 1410B, etc., may include head-up displays, terminals, personal computers, game consoles, tablets, telephones, set-top boxes, kiosks, wireless devices, digital mats, standalone devices, handheld gaming devices, etc. Typically, client 1410 is configured to receive encoded video streams, decode video streams, and present the resulting video to a user, such as a game player. The process of receiving and / or decoding encoded video streams typically involves storing individual video frames in the client's receive buffer. The video stream may be presented to the user on a display integrated with client 1410 or on a separate device, such as a monitor or television. Client 1410 is optionally configured to support more than one game player. For example, a game console may be configured to support two, three, four, or more simultaneous players. Each of these players may receive a separate video stream, or a single video stream may include a region of frames specifically generated for each player, for example, a region of frames generated based on each player's viewpoint. Client 1410 is optionally geographically distributed. The number of clients included in the gaming system 1400 can vary widely from one or two to thousands, tens of thousands, or more. As used herein, the term "gamer" refers to a person playing a game, and the term "gaming device" refers to a device used to play a game. In some embodiments, a gaming device can refer to multiple computing devices that collaborate to deliver a gaming experience to a user. For example, a game console and an HMD can collaborate with the video server system 1420 to deliver a game viewed through the HMD. In one embodiment, the game console receives a video stream from the video server system 1420, and the game console forwards the video stream or updates to the video stream to the HMD for rendering.

[0142] Client 1410 is configured to receive a video stream via network 1415. Network 1415 can be any type of communication network, including telephone networks, the Internet, wireless networks, power line networks, local area networks, wide area networks, private networks, etc. In a typical implementation, the video stream is transmitted via standard protocols such as TCP / IP or UDP / IP. Alternatively, the video stream is transmitted via proprietary standards.

[0143] A typical example of client 1410 is a personal computer, including a processor, non-volatile memory, a display, decoding logic, network communication functions, and input devices. The decoding logic may include hardware, firmware, and / or software stored on a computer-readable medium. Systems for decoding (and encoding) video streams are well known in the art and vary depending on the specific encoding scheme used.

[0144] Client 1410 may, but is not required to, further include a system configured to modify the received video. For example, the client may be configured to perform further rendering to overlay one video image onto another, crop video images, etc. For example, client 1410 may be configured to receive various types of video frames, such as I-frames, P-frames, and B-frames, and process these frames into images for display to the user. In some embodiments, components of client 1410 are configured to perform further rendering, colorization, conversion to 3D, or similar operations on the video stream. Components of client 1410 may optionally be configured to receive more than one audio or video stream. Input devices for client 1410 may include, for example, a one-handed game controller, a two-handed game controller, a gesture recognition system, a gaze recognition system, a voice recognition system, a keyboard, a joystick, an indicator, a force feedback device, a motion and / or position sensing device, a mouse, a touchscreen, a neural interface, a camera, and other input devices yet to be developed.

[0145] A video stream (and optionally an audio stream) is generated and provided by the video server system 1420 and received by the client 1410. As further described elsewhere herein, this video stream includes video frames (and the audio stream includes audio frames). Video frames (e.g., the video frames include pixel information in a suitable data structure) are configured to meaningfully contribute to an image displayed to a user. As used herein, the term "video frame" is used to refer primarily to a frame that includes information configured to contribute to, for example, realize, an image displayed to a user. Much of the teaching herein regarding "video frames" can also be applied to "audio frames."

[0146] Client 1410 is typically configured to receive input from a user. This input may include game commands configured to change the state of the video game or otherwise affect the game's plot. Game commands can be received using input devices and / or can be automatically generated by computational instructions executed on client 1410. The received game commands are transmitted from client 1410 to video server system 1420 and / or game server 1425 via network 1415. For example, in some embodiments, game commands are transmitted to game server 1425 via video server system 1420. In some embodiments, separate copies of the game commands are transmitted from client 1410 to game server 1425 and video server system 1420. The transmission of game commands optionally depends on the identifier of the command. Game commands are optionally transmitted from client 1410A via different routes or communication channels used to provide audio or video streams to client 1410A.

[0147] Game server 1425 is optionally operated by an entity different from video server system 1420. For example, game server 1425 may be operated by a publisher of a multiplayer game. In this example, video server system 1420 is optionally regarded as a client by game server 1425 and is optionally configured to appear from the viewpoint of game server 1425 as a prior art client executing a prior art game engine. Communication between video server system 1420 and game server 1425 optionally occurs via network 1415. Thus, game server 1425 may be a prior art multiplayer game server that sends game status information to multiple clients, one of which is game server system 1420. Video server system 1420 may be configured to communicate with multiple instances of game server 1425 simultaneously. For example, video server system 1420 may be configured to serve multiple different video games to different users. Each of these different video games may be supported by a different game server 1425 and / or published by a different entity. In some implementations, several geographically distributed instances of the video server system 1420 are configured to serve game videos to multiple different users. Each of these instances of the video server system 1420 can communicate with the same instance of the game server 1425. Communication between the video server system 1420 and one or more game servers 1425 optionally occurs via a dedicated communication channel. For example, the video server system 1420 may connect to the game server 1425 via a high-bandwidth channel dedicated to communication between the two systems.

[0148] The video server system 1420 includes at least a video source 1430, I / O devices 1445, a processor 1450, and non-transitory storage devices 1455. The video server system 1420 may include a single computing device or be distributed among multiple computing devices. These computing devices are optionally connected via a communication system, such as a local area network (LAN).

[0149] Video source 1430 is configured to provide a video stream, such as streaming video or a series of video frames forming motion graphics. In some embodiments, video source 1430 includes a video game engine and rendering logic. The video game engine is configured to receive game commands from players and maintain a copy of the video game state based on the received commands. This game state includes the positions of objects in the game environment and the usual viewpoint. The game state may also include object attributes, images, colors, and / or textures. The game state is typically maintained based on game rules and game commands such as move, turn, attack, set focus, interact, use, etc. Optionally, a portion of the game engine is housed within game server 1425. Game server 1425 can maintain copies of the game state using geographically dispersed clients based on game commands received from multiple players. In these cases, the game state is provided by game server 1425 to video source 1430, where copies of the game state are stored and rendering is performed. Game server 1425 may receive game commands directly from client 1410 via network 1415 and / or may receive game commands via video server system 1420.

[0150] Video source 1430 typically includes rendering logic, such as hardware, firmware, and / or software, stored on a computer-readable medium, such as storage device 1455. This rendering logic is configured to create video frames for a video stream based on the game state. All or part of the rendering logic is optionally located within a graphics processing unit (GPU). The rendering logic typically includes a processing stage configured to determine three-dimensional spatial relationships between objects and / or apply appropriate textures, etc., based on the game state and viewpoint. The rendering logic produces raw video, which is then typically encoded before being transmitted to client 1410. For example, the raw video may be encoded according to Adobe Flash® standards, .wav, H.264, H.263, On2, VP6, VC-1, WMA, Huffyuv, Lagarith, MPG-x, Xvid, FFmpeg, x264, VP6-8, realvideo, mp3, etc. The encoding process produces a video stream, which is optionally packaged for delivery to a decoder on a remote device. The video stream is characterized by its frame size and frame rate. Typical frame sizes include 800×600, 1280×720 (e.g., 720p), and 1024×768, but any other frame size can be used. Frame rate is the number of video frames per second. Video streams can include different types of video frames. For example, the H.264 standard includes "P" frames and "I" frames. I-frames contain information used to refresh all macroblocks / pixels on the display device, while P-frames contain information used to refresh a subset of them. P-frames are typically smaller than I-frames. As used herein, the term "frame size" refers to the number of pixels within a frame. The term "frame data size" refers to the number of bytes required to store a frame.

[0151] In an alternative embodiment, video source 1430 includes a video recording device, such as a camera. This camera can be used to generate delayed or live video that may be included in a video stream of a computer game. The resulting video stream optionally includes rendered images and images recorded using a still camera or video camera. Video source 1430 may also include a storage device configured to store previously recorded video for inclusion in the video stream. Video source 1430 may further include: a motion or position sensing device configured to detect the movement or position of an object, such as a person; and logic configured to determine a game state or generate video based on the detected motion and / or position.

[0152] Video source 1430 is optionally configured to provide overlays, which are configured to be placed on top of another video. For example, these overlays may include a command interface, login instructions, messages to game players, images of other game players, or video feeds from other game players (e.g., webcam video). In embodiments of client 1410A that include a touchscreen interface or a gaze-oriented interface, the overlay may include a virtual keyboard, joystick, touchpad, etc. In one instance of overlay, the player's voice is overlaid on an audio stream. Video source 1430 may optionally also include one or more audio sources.

[0153] In an implementation where the video server system 1420 is configured to maintain the game state based on input from more than one player, each player can have a different viewpoint, including view position and orientation. The video source 1430 is optionally configured to provide a separate video stream for each player based on their viewpoint. Furthermore, the video source 1430 can be configured to provide different frame sizes, frame data sizes, and / or encodings to each client 1410. The video source 1430 is optionally configured to provide 3D video.

[0154] I / O device 1445 is configured for use in video server system 1420 to send and / or receive information such as video, commands, requests for information, game status, gaze information, device motion, device position, user motion, client representation, player identity, game commands, security information, audio, etc. I / O device 1445 typically includes communication hardware such as a network interface card (NIC) or modem. I / O device 1445 is configured to communicate with game server 1425, network 1415, and / or client 1410.

[0155] Processor 1450 is configured to execute logic, such as software, in various components of the video server system 1420 discussed herein. For example, processor 1450 can be programmed via software instructions to perform the functions of video source 1430, game server 1425, and / or client limiter 1460. Video server system 1420 optionally includes more than one instance of processor 1450. Processor 1450 can also be programmed via software instructions to execute commands received by video server system 1420 or to coordinate the operation of various elements of game system 1400 discussed herein. Processor 1450 may include one or more hardware devices. Processor 1450 is an electronic processor.

[0156] Storage device 1455 includes non-transitory analog and / or digital storage devices. For example, storage device 1455 may include an analog storage device configured to store video frames. Storage device 1455 may include a computer-readable digital storage device, such as a hard disk drive, optical drive, or solid-state storage device. Storage device 1455 is configured (e.g., via a suitable data structure or file system) to store video frames, artificial frames, video streams including both video frames and artificial frames, audio frames, audio streams, etc. Storage device 1455 may optionally be distributed among multiple devices. In some embodiments, storage device 1455 is configured to store software components of the video source 1430 discussed elsewhere herein. These components may be stored in a readily available format when needed.

[0157] The video server system 1420 optionally also includes a client limiter 1460. The client limiter 1460 is configured to remotely determine the capabilities of a client, such as client 1410A or 1410B. These capabilities may include the capabilities of client 1410A itself, and the capabilities of one or more communication channels between client 1410A and the video server system 1420. For example, the client limiter 1460 may be configured to test the communication channels via network 1415.

[0158] Client qualifier 1460 can manually or automatically determine (e.g., discover) the capabilities of client 1410A. Manual determination includes communicating with the user of client 1410A and requesting the user to provide capabilities. For example, in some embodiments, client qualifier 1460 is configured to display images, text, etc., within the browser of client 1410A. In one embodiment, client 1410A is an HMD including a browser. In another embodiment, client 1410A is a game console with a browser that can be displayed on the HMD. The displayed object requests user input information, such as the operating system, processor, video decoder type, network connection type, display resolution, etc., of client 1410A. The user-input information is transmitted back to client qualifier 1460.

[0159] For example, this can be automatically determined by executing an agent on client 1410A and / or by sending a test video to client 1410A. The agent may include computational instructions, such as JavaScript, embedded in a webpage or installed as an add-on. The agent is optionally provided by client qualifier 1460. In various implementations, the agent may discover the processing capabilities of client 1410A, the decoding and display capabilities of client 1410A, the latency reliability and bandwidth of the communication channel between client 1410A and video server system 1420, the display type of client 1410A, firewalls present on client 1410A, the hardware of client 1410A, software executing on client 1410A, registry entries within client 1410A, etc.

[0160] The client qualifier 1460 includes hardware, firmware, and / or software stored on a computer-readable medium. The client qualifier 1460 is optionally located on a computing device separate from one or more other components of the video server system 1420. For example, in some embodiments, the client qualifier 1460 is configured to determine the characteristics of a communication channel between client 1410 and more than one instance of the video server system 1420. In these embodiments, the information discovered by the client qualifier can be used to determine which instance of the video server system 1420 is best suited to deliver streaming video to a client 1410.

[0161] The embodiments of this disclosure can be practiced with various computer system configurations, including handheld devices, microprocessor systems, microprocessor-based or programmable consumer electronic devices, minicomputers, mainframes, etc. This disclosure can also be practiced in distributed computing environments, where tasks are performed by remote processing devices via wired or wireless network links.

[0162] In light of the above embodiments, it should be understood that this disclosure can employ various computer-implemented operations involving data stored in a computer system. These operations are those requiring physical manipulation of physical quantities. Any operation described herein that forms part of this disclosure is a useful machine operation. This disclosure also relates to means or apparatus for performing these operations. The apparatus may be specifically constructed for the desired purpose, or the apparatus may be a general-purpose computer selectively activated or configured by a computer program stored in a computer. Specifically, various general-purpose machines may be used with computer programs written according to the teachings herein, or more specialized apparatus may be more readily constructed to perform the desired operations.

[0163] This disclosure can also be implemented as computer-readable code on a computer-readable medium. A computer-readable medium is any data storage device capable of storing data that can subsequently be read by a computer system. Examples of computer-readable media include hard disk drives, network-attached storage (NAS), read-only memory, random access memory, CD-ROMs, CD-Rs, CD-RWs, magnetic tapes, and other optical and non-optical data storage devices. Computer-readable media may also include computer-readable tangible media distributed across a network-coupled computer system, such that computer-readable code is stored and executed in a distributed manner.

[0164] Although the method operations are described in a specific order, it should be understood that other housekeeping operations may be performed between operations, or operations may be adjusted so that they occur at slightly different times, or they may be distributed in a system that allows processing operations to be sent at various intervals associated with the processing, as long as the processing of the superimposed operations is performed in the desired manner.

[0165] Although the foregoing disclosure has been described in some detail for clarity of understanding, it will be apparent that certain changes and modifications may be practiced within the scope of the appended claims. Therefore, embodiments of the invention are to be regarded as illustrative rather than restrictive, and this disclosure is not limited to the details given herein, but may be modified within the scope of this disclosure and its equivalents.

Claims

1. A method, the method comprising: Record game plot metadata generated from the execution of a first session of a video game, the execution of which is driven by a first user using a first head-up display through an interactive game plot of the video game, wherein the execution of the first session causes a first view of the virtual environment of the video game to be rendered for presentation via the first head-up display, the first view being derived from a first position in the virtual environment determined by the interactive game plot, and the first view also being based on tracked movement of the first head-up display. After the first session is completed, the game plot metadata is transmitted to the client device; The client device tracks the movement of the second head-up display; The client device executes a second session of the video game using the game plot metadata to reconstruct game plot events from the first session in the second session, wherein the execution of the second session causes a second view of the virtual environment to be rendered for presentation via a second head-up display, the second view being derived from a second position in the virtual environment, the second position being determined based on a first position in the virtual environment, the second position moving in response to movement of the first position to maintain a predefined spatial relationship between the first position and the second position in the virtual environment.

2. The method of claim 1, wherein the movement of the second position in order to maintain the predefined spatial relationship does not exceed a predefined maximum speed in the virtual environment.

3. The method of claim 1, wherein the movement of the second position in order to maintain the predefined spatial relationship does not exceed a predefined maximum acceleration in the virtual environment.

4. The method of claim 1, wherein the movement of the second position in order to maintain the predefined spatial relationship allows the second position to drift from the predefined spatial relationship while continuously tracking and approaching the predefined spatial relationship.

5. The method of claim 1, wherein the movement of the second position in order to maintain the predefined spatial relationship is linearly interpolated based on the current spatial position of the first position to track proximity to realize the predefined spatial relationship.

6. The method of claim 5, wherein the linear interpolation follows a path that minimizes the amount of time required to achieve the predefined spatial relationship.

7. The method of claim 1, wherein the game plot metadata includes game state values ​​generated by the execution of the first session of the video game.

8. The method of claim 1, wherein the first session is performed by a computing device remote from the client device, the computing device and the client device being connected to a network, and the game plot metadata being transmitted through the network.

9. A method, the method comprising: Record game plot metadata generated from the execution of a first session of a video game, the execution of which is driven by a first user using a first head-up display through an interactive game plot of the video game, wherein the execution of the first session causes a first view of the virtual environment of the video game to be rendered for presentation via the first head-up display, the first view being derived from a first orientation in the virtual environment determined by the interactive game plot, and the first view also being based on tracked movement of the first head-up display. After the first session is completed, the game plot metadata is transmitted to the client device; The client device tracks the movement of the second head-up display; The client device executes a second session of the video game using the game plot metadata to reconstruct game plot events from the first session in the second session, wherein the execution of the second session causes a second view of the virtual environment to be rendered for presentation via a second head-up display, the second view having a second orientation in the virtual environment, the second orientation being determined based on the first orientation in the virtual environment, the second orientation changing in response to a change in the first orientation to maintain a predefined spatial relationship between the first orientation and the second orientation in the virtual environment.

10. The method of claim 9, wherein the change in the second orientation for maintaining the predefined spatial relationship does not exceed a predefined maximum speed in the virtual environment.

11. The method of claim 9, wherein the change in the second orientation for maintaining the predefined spatial relationship does not exceed a predefined maximum acceleration in the virtual environment.

12. The method of claim 9, wherein the change in the second orientation for maintaining the predefined spatial relationship allows the second orientation to drift from the predefined spatial relationship while continuously tracking proximity to the predefined spatial relationship.

13. The method of claim 9, wherein the change in the second orientation for maintaining the predefined spatial relationship is linearly interpolated based on the current position of the first orientation to track proximity to achieve the predefined spatial relationship.

14. The method of claim 13, wherein the linear interpolation is configured to minimize the amount of time required to achieve the predefined spatial relationship.

15. The method of claim 14, wherein the game plot metadata includes game state values ​​generated by the execution of the first session of the video game.

16. The method of claim 9, wherein the first session is performed by a computing device remote from the client device, the computing device and the client device being connected to a network, and the game plot metadata being transmitted through the network.

17. A method, the method comprising: Record game plot metadata generated from the execution of a first session of a video game, the execution of which is driven by a first user using a first head-up display through an interactive game plot of the video game, wherein the execution of the first session causes a first view of the virtual environment of the video game to be rendered for presentation via the first head-up display, the first view being derived from a first position and orientation in the virtual environment determined by the interactive game plot, and the first view also being based on tracked movement of the first head-up display; After the first session is completed, the game plot metadata is transmitted to the client device; The client device tracks the movement of the second head-up display; The client device executes a second session of the video game using the game plot metadata to reconstruct game plot events from the first session in the second session, wherein the execution of the second session causes a second view of the virtual environment to be rendered for presentation via a second head-up display. The second view is derived from a second position and orientation in the virtual environment, the second position and orientation being determined based on the first position and orientation in the virtual environment. The second position and orientation moves in response to movement of the first position and orientation to maintain a predefined spatial relationship between the first position and orientation and the second position and orientation in the virtual environment.

18. The method of claim 17, wherein the movement of the second position and orientation in order to maintain the predefined spatial relationship does not exceed a predefined maximum speed in the virtual environment.

19. The method of claim 17, wherein the movement of the second position and orientation in order to maintain the predefined spatial relationship does not exceed a predefined maximum acceleration in the virtual environment.

20. The method of claim 17, wherein the movement of the second position and orientation in order to maintain the predefined spatial relationship allows the second position and orientation to drift from the predefined spatial relationship while continuously tracking and approaching the predefined spatial relationship.

Images