Field Mapping for Virtual Reality Viewing of Esports

Abstract

In some implementations, a method is provided, the method comprising the following operations: receiving a request from a virtual reality viewer to watch a live event through a head-mounted display from a client device over a network; assigning the virtual reality viewer to a seat in a venue where the live event is occurring; receiving a plurality of video feeds from a plurality of cameras positioned in the venue; accessing video processing parameters stored in association with the seats; using the video processing parameters to select a video feed from the video feeds and stitching the selected video feeds to generate a composite video, the composite video providing a view of the venue from an angle substantially defined by a 3D position of the seat in the venue; transmitting the composite video to the client device over the network for rendering to the head-mounted display.

Description

1. Technical Field

[0002] The present disclosure relates to venue mapping for virtual reality viewing of esports. Background Art 2. Background technology

[0004] Electronic sports (e-sports) generally refers to a form of competition in which the main aspects of the competition are facilitated by electronic systems, where the input of players and teams and the output of the e-sports system are transmitted through a human-computer interface. (See, for example, Juho Hamari, Max (2017) "What is eSports and why do people watch it?", Internet Research, Vol. 27, No. 2, pp. 211-232, which is incorporated herein by reference). In practice, eSports encompasses competitive and professional video game events that are viewed. For example, but not limitation, eSports may be viewed live (e.g., at a competition venue) via online broadcast or online streaming, as well as via television broadcast. Many eSports events take the form of organized leagues that feature multiplayer video game competitions, particularly competitions between teams of players that may include amateur and professional players. Common video game genres associated with eSports include real-time strategy games (RTS), fighting games, first-person shooters (FPS), and multiplayer online battle arenas (MOBAs).

[0005] Video games are played by computing devices such as personal computers, game consoles, mobile devices, etc. An example of a gaming platform is the Sony (PS4), which is sold as a game console. As is well known, game consoles are designed to be connected to a display (usually a television) and enable user interaction through a handheld controller. Game consoles are designed with specialized processing hardware, including a CPU, a graphics synthesizer for handling intensive graphics operations, a vector unit for performing geometric transformations, and other glue hardware, firmware, and software. Game consoles may be further designed with a disc reader, which is used to receive game discs for local games through the game console. Online gaming is also possible, where users can play games interactively with other users over the Internet. As game complexity continues to attract players, game and hardware manufacturers continue to innovate to enable additional interactivity.

[0006] A growing trend in the computer gaming industry is to develop games that increase the interaction between the user and the gaming system. One way to achieve a richer interactive experience is to use a wireless game controller whose movements are tracked by the gaming system so that the player's movements are tracked and used as input for the game. Generally speaking, gesture input refers to making an electronic device such as a computing system, video game console, smart appliance, etc. react to certain gestures made by the player and captured by the electronic device.

[0007] Another way to achieve a more immersive interactive experience is to use a head-mounted display (HMD). A head-mounted display is worn by a user and can be configured to present various graphics, such as a view of a virtual space. The graphics presented on the head-mounted display can cover most or even all of the user's field of view. Thus, a head-mounted display can provide a visually immersive virtual reality experience for the user because the HMD renders a three-dimensional real-time view of the virtual environment in a manner that responds to the user's movements. A user wearing the HMD can move freely in all directions, and thus a view of the virtual environment can be provided in all directions via the HMD.

[0008] Against this background, the implementation of the present disclosure is produced. Summary of the invention

[0009] Implementations of the present disclosure include apparatus, methods, and systems related to venue mapping for virtual reality viewing of e-sports.

[0010] In some implementations, a method is provided, the method comprising the following operations: receiving a request from a virtual reality viewer to watch a live event through a head-mounted display from a client device over a network; assigning the virtual reality viewer to a seat in a venue where the live event is occurring; receiving a plurality of video feeds from a plurality of cameras positioned in the venue; accessing video processing parameters stored in association with the seats; using the video processing parameters to select a video feed from the video feeds and stitching the selected video feeds to generate a composite video, the composite video providing a view of the venue from an angle substantially defined by a 3D position of the seat in the venue; transmitting the composite video to the client device over the network for rendering to the head-mounted display.

[0011] In some implementations, the video processing parameters identify which of the video feeds are selected for stitching, the video processing parameters being defined based on the 3D position of the seat associated with the video processing parameters and a 3D position of the camera providing the video feed.

[0012] In some implementations, assigning the virtual reality viewer to the seat includes: identifying an occupancy status of seats in the venue, wherein the occupancy status of a given seat indicates whether the given seat is occupied by a real viewer in the venue, and wherein the seat assigned to the virtual reality viewer is a seat that is not occupied by a real viewer.

[0013] In some implementations, the occupancy status of the given seat further indicates whether the given seat is occupied by another virtual reality viewer, wherein the seat allocated to the virtual reality viewer is a seat that is not occupied by another virtual reality viewer.

[0014] In some implementations, assigning the virtual reality viewer to the seat includes: accessing a social graph of the virtual reality viewer; and selecting the seat based on proximity to a seat assigned to another virtual reality viewer who is a member of the social graph.

[0015] In some implementations, the method further includes: accessing audio processing parameters stored in association with a seat; using the audio processing parameters to generate audio data that simulates listening from an angle substantially defined by the 3D position of the seat in the venue; and transmitting the audio data to the client device over the network.

[0016] In some implementations, the audio processing parameters identify audio captured by one or more microphones in the venue to generate the audio data from the audio, and the audio processing parameters are defined based on the 3D positions of the seats and the 3D positions of the microphones associated with the audio processing parameters.

[0017] In some implementations, a non-transitory computer-readable medium is provided, the non-transitory computer-readable medium having program instructions embodied thereon, the program instructions, when executed by at least one computer, causing the at least one computer to perform a method comprising: receiving a request from a virtual reality spectator to view a live event through a head-mounted display from a client device over a network; assigning the virtual reality spectator to a seat in a venue where the live event is occurring; receiving a plurality of video feeds from a plurality of cameras positioned in the venue; accessing video processing parameters stored in association with the seats; using the video processing parameters to select a video feed from the video feeds and stitching the selected video feeds to generate a composite video, the composite video providing a view of the venue from an angle substantially defined by a 3D position of the seat in the venue; transmitting the composite video to the client device over the network for rendering to the head-mounted display.

[0018] In some implementations, a system is provided that includes at least one computing device having at least one processor and at least one memory, the at least one computing device being configured to: receive a request from a client device over a network for a virtual reality spectator to view a live event through a head-mounted display; assign the virtual reality spectator to a seat in a venue where the live event is occurring; receive a plurality of video feeds from a plurality of cameras positioned in the venue; access video processing parameters stored in association with the seats; use the video processing parameters to select a video feed from the video feeds and stitch the selected video feeds to generate a composite video that provides a view of the venue from an angle substantially defined by a 3D position of the seat in the venue; and transmit the composite video to the client device over the network for rendering to the head-mounted display.

[0019] Other aspects and advantages of the present disclosure will become apparent from the following detailed description, taken in conjunction with the accompanying drawings, illustrating by way of example the principles of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] The present disclosure may be better understood by reference to the following detailed description in conjunction with the accompanying drawings, in which:

[0021] Figure 1A A diagram of an electronic sports (e-sports) venue is shown in accordance with implementations of the present disclosure.

[0022] Figure 1B is a conceptual top view of a venue according to an implementation of the present disclosure.

[0023] Figure 1C A portion of the seating of a venue where a live eSports event occurs is conceptually illustrated in accordance with implementations of the present disclosure.

[0024] Figure 2 A process for mapping the three-dimensional space of an eSports venue to determine seat-specific parameters for virtual reality viewing in accordance with implementations of the present disclosure is conceptually illustrated.

[0025] Figure 3 A virtual reality viewer's field of view is conceptually illustrated according to implementations of the present disclosure.

[0026] Figure 4 A system for providing virtual reality viewing of an e-sports event according to an implementation of the present disclosure is conceptually illustrated.

[0027] Figure 5 Techniques for determining whether seats in a venue are occupied by real spectators according to implementations of the present disclosure are shown.

[0028] Figure 6 A method for a virtual reality viewer to see himself / herself in the context of a real venue according to an implementation of the present disclosure is shown.

[0029] Figure 7 Seats in a venue with additional functionality for sensing real spectators and enabling interactivity for virtual reality spectators are shown in accordance with implementations of the present disclosure.

[0030] Figure 8 A system for interacting with a virtual environment via a head mounted display (HMD) according to implementations of the present disclosure is shown.

[0031] Figure 9A-1 and Figure 9A-2 A head mounted display (HMD) according to implementations of the present disclosure is shown.

[0032] Fig. 9B An example is shown of an HMD user interfacing with a client system and the client system providing content to a second screen display (referred to as the second screen) according to one implementation.

[0033] Fig.10 The functionality of an HMD according to implementations of the present disclosure is conceptually illustrated in conjunction with an executing video game.

[0034] Fig.11 Components of a head mounted display according to implementations of the present disclosure are shown.

[0035] Fig.12 is a block diagram of a gaming system 1200 according to various implementations of the present disclosure. DETAILED DESCRIPTION

[0036] The following implementations of the present disclosure provide apparatus, methods, and systems related to field mapping for virtual reality viewing of e-sports. However, it will be apparent to those skilled in the art that the present disclosure may be practiced without some or all of the specific details currently described. In other cases, well-known processing operations are not described in detail in order not to unnecessarily obscure the present invention.

[0037] In various implementations, methods, systems, image capture objects, sensors, and associated interface objects (e.g., controllers, gloves, peripheral devices, etc.) are configured to process data that is configured to be rendered on a display screen substantially in real time. Broadly speaking, implementations are described with reference to a display that is a head mounted display (HMD). However, in other implementations, the display may be a second screen, a display of a portable device, a computer display, a display panel, a display of one or more remotely connected users (e.g., users who may be viewing content or sharing in an interactive experience), etc.

[0038] Figure 1A A view of an electronic sports (e-sports) venue according to an implementation of the present disclosure is shown. E-sports generally refers to competitive or professional games, particularly multi-player video games, that are watched by various spectators or users. With the popularity of e-sports in recent years, there has been an increasing interest in watching e-sports events live at physical venues, many of which can seat thousands of people. A suitable venue can be any location where an e-sports event can be held for live viewing by spectators, including but not limited to, for example, an arena, stadium, theater, convention center, gymnasium, community center, etc.

[0039] However, holding and staging e-sports events such as leagues at inconspicuous physical venues means that not everyone who wishes to watch live can attend. Therefore, it is desirable to provide a live experience to remote viewers so that the remote viewer can experience the e-sports event as if he / she were physically present at the venue where the e-sports event is taking place.

[0040] Continue to refer Figure 1A , a view of a venue 100 where an e-sports event is being held is shown. A typical e-sports event is a tournament in which teams of players compete against each other in a multi-player video game. In the illustrated implementation, a first team is composed of players 102a, 102b, 102c, and 102d, and a second team is composed of players 104a, 104b, 104c, and 104d. The first team and the second team are located on a stage 106 along with an announcer / host 110. The first team and the second team participate in competitive game play of a multi-player video game against each other at venue 100, and spectators 113 are present to watch the event.

[0041] Large displays 108a, 108b, and 108c provide a view of the gameplay to the audience 113. It should be understood that the displays 108a, 108b, and 108c can be any type of display known in the art that can present the content of the gameplay to the audience, including but not limited to, for example, LED displays, LCD displays, DLP, etc. In some implementations, the displays 108a, 108b, and 108c are display screens to which one or more projectors (not shown) project the video / image of the gameplay. It should be understood that the displays 108a, 108b, and 108c can be configured to present any of a variety of content, including but not limited to, for example, the content of the gameplay, the player's view of the video game, the game map, the audience's view of the video game, the commentator's view, the statistics and scores of the players / teams, advertisements, etc.

[0042] In addition, commentators 112a and 112b provide commentary regarding the game play, such as describing the game play in real time as the game play progresses, providing analysis of the game play, highlighting certain activities, etc.

[0043] Figure 1B 1 is a conceptual top view of a venue 100 according to an implementation of the present disclosure. As previously described, players of a first team and a second team are located on a stage and participate in the game play of a multiplayer video game. A plurality of seats 114 are conceptually shown that may be occupied by spectators when they attend and watch an e-sports event in person. As described, there are large displays 108a, 108b, and 108c that provide views of the game play and other content for spectators 113 to view. In addition, there are a number of speakers 118 that may be distributed throughout the venue to provide audio for spectators to hear, including audio associated with or related to any content presented on displays 108a, 108b, and 108c.

[0044] In addition, according to the implementation of the present disclosure, there are any number of cameras 116 distributed throughout the venue 100, and the cameras 116 are configured to capture video of the e-sports event for processing, distribution, streaming and / or viewing by spectators in person and / or remote spectators. It should be understood that some cameras 116 may have a fixed position and / or orientation, while some cameras 116 may have a variable position and / or orientation and may be able to be moved to a new position and / or reoriented to a new direction. It should be understood that the camera 116 may have various fields of view. In addition, some cameras may be 360-degree cameras capable of capturing a 360-degree field of view (e.g., a 360-degree horizontal field of view, combined with a 180-degree vertical field of view, which can provide a complete spherical field of view). Such 360-degree cameras typically include multiple image capture devices in a single device package. In some implementations, multiple cameras are configured in substantially or approximately the same position to achieve a 360-degree field of view from the perspective of the position when their feeds are spliced ​​together.

[0045] According to implementations of the present disclosure, a “live” viewing experience of an e-sports event may be provided to a virtual reality spectator 120. That is, a view is provided to the virtual reality spectator 120 through a head mounted display (HMD) (or a virtual reality headset) that simulates the experience of being in attendance at an e-sports event and occupying a particular seat 122 (or designated position) at the venue 100. Broadly speaking, the three-dimensional (3D) position of the virtual reality spectator's seat 122 may be determined, and video feeds from certain of the various cameras 116 may be stitched together to provide a virtual reality view of the venue 100 from the perspective of the seat 122 (or designated position assigned to the virtual reality spectator).

[0046] Furthermore, although not specifically shown, each camera may include at least one microphone for capturing audio from venue 100. Additionally, there may be additional microphones distributed throughout venue 100. Audio from at least some of these microphones may also be processed based on the 3D position of the virtual reality viewer's seat 122 to provide audio that simulates the audio that would be heard from the perspective of a person occupying seat 122.

[0047] Figure 1CA portion 124 of seats of a venue 100 where a live e-sports event takes place is conceptually illustrated in accordance with an implementation of the present disclosure. As shown, a view simulating the occupation of seats 122 in the venue 100 is presented to a virtual reality spectator 120 via an HMD 150. In some implementations, the view of the e-sports event provided to the virtual reality spectator 120 is provided from a streaming service 142 via a network 144. That is, the streaming service 142 includes one or more server computers configured to stream video for rendering on the HMD 150, wherein the rendered video provides a view of the e-sports event to the virtual reality spectator 120. Although not specifically illustrated in the illustrated implementation, it should be understood that the streaming service 142 may first transmit the video in the form of data via the network 144 to a computing device local to the virtual reality spectator 120, wherein the computing device may process the data for rendering to the HMD 150.

[0048] The streaming service 142 may provide an interface to the virtual reality viewer 120 that enables the virtual reality viewer 120 to select or subscribe to one or more views to be streamed for rendering on the HMD 150. As described, these views may be 360-degree views of the event / venue from the perspective of a particular seat or position in the venue to provide an immersive viewing experience to the virtual reality viewer 120.

[0049] It should be understood that the provided view responds in real time to the movement of the virtual reality spectator 120, for example, so that if the virtual reality spectator 120 turns left, the virtual reality spectator 120 (through the HMD 150) sees a view to the left of the seat 122, and if the virtual reality spectator 120 turns right, the virtual reality spectator 120 (through the HMD 150) sees a view to the right of the seat 122, and so on. In some implementations, the virtual reality spectator 120 is provided with a potential view of the e-sports venue 100 in all directions (including a 360-degree horizontal field of view). In some implementations, the virtual reality spectator 120 is provided with a potential view of the e-sports venue 100 in a subset of all directions (such as a horizontal field of view of approximately 270 degrees in some implementations, or a horizontal field of view of 180 degrees in some implementations). In some implementations, the provided field of view may exclude areas directly above or directly below. In some implementations, other content, such as advertisements, splash screens, logo content, game-related images or videos, etc., may be provided to areas excluded from the field of view of the e-sports venue.

[0050] In some implementations, virtual reality spectators 120 are able to select seats through the interface so that they can watch the e-sports event from the perspective of their choice. In some implementations, the selectable seats are seats that are not actually occupied by spectators who are in attendance at the e-sports event. In other implementations, both unoccupied seats and occupied seats are selectable for virtual reality viewing.

[0051] In some implementations, the streaming service 142 can automatically assign the virtual reality viewer to a specific seat. In some implementations, this may be the best available seat (e.g., according to a predefined order or ranking of available seats).

[0052] In some implementations, the virtual reality viewer can be assigned to a seat that is close to other viewers based on various characteristics of the viewer. In some implementations, the virtual reality viewer is assigned to a seat based at least in part on the virtual reality viewer's membership in a social network / social graph. For example, referring to Figure 1C , another VR viewer 132 may be a friend of VR viewer 120 on social network 130 (e.g., as defined by membership in a social graph). Streaming service 142 may use this information to assign VR viewer 132 to a seat that is proximal to VR viewer 120, such as seat 134 that is adjacent to seat 122 already assigned to VR viewer 120. In the illustrated implementation, thus being virtually “seated,” when VR viewer 120 turns to the right, VR viewer 120 may see the avatar of VR viewer 132 seated next to him.

[0053] In some implementations, the interface for seat selection and / or assignment may inform a given user that one or more of their friends on social network 130 are also virtually attending the esports event, and provide the option to automatically be assigned a seat near one or more of their friends. In this way, friends attending the same event as the virtual reality viewer can enjoy the event together.

[0054] In various implementations, virtual reality viewers may be assigned to seats that are proximate to one another based on any of a variety of factors, such as user profile, age, geographic location, primary language, experience in a given video game, interests, gender, etc.

[0055] In some implementations, these concepts may be expanded to include live "real" audience members (who are actually in attendance, not virtual reality audience members) when information about such real audience members is known. For example, it may be determined that real audience member 138 sitting in seat 136 is a friend of virtual reality audience member 120, and thus virtual reality audience member 120 may be assigned (or offered to be assigned) to seat 122 adjacent to seat 136 of real audience member 138.

[0056] It should be understood that in order to provide input while viewing content through the HMD, the virtual reality spectator can use one or more controller devices. In the illustrated implementation, the virtual reality spectators 120, 132, and 140 operate the controller devices 152, 156, and 160, respectively, to provide input to (for example) start and stop the streaming of the video for virtual reality viewing, select a seat for viewing in the e-sports venue 100, etc.

[0057] It should be appreciated that, whether virtual or real, spectators, in some implementations, can hear each other if they are in close proximity to one another in the esports venue 100. For example, virtual reality spectators 120 and 132 can hear each other's voices when audio captured from their respective local environments (e.g., via microphones in HMDs 150 and 154, controllers 152 and 156, or elsewhere in the local environments of spectators 120 and 132) is provided to the other's video streams. In some implementations, virtual reality spectator 120 can hear the voices from real spectator 138 captured by local microphones.

[0058] In some implementations, virtual reality viewers may occupy seats that are actually occupied by real viewers. For example, in the illustrated implementation, virtual reality viewers may occupy seats 136 that are actually occupied by real viewers 138. Virtual reality viewers 140 are provided with a view that simulates being in the position of seat 136. When virtual reality viewers 140 turn right, they can see the avatar of viewer 120; similarly, when viewer 120 turns left, they can see the avatar of viewer 140 instead of real viewer 138.

[0059] Figure 2 A process for mapping the three-dimensional space of an esports venue to determine seat-specific parameters for virtual reality viewing is conceptually illustrated in accordance with an implementation of the present disclosure. A three-dimensional (3D) space 200 of the venue 100 is illustrated on the left side of the illustrated implementation. In accordance with an implementation of the present disclosure, as shown at reference numeral 210, a 3D spatial mapping of the venue space 200 may be generated to facilitate determining seat-specific parameters for virtual reality viewing.

[0060] Various techniques may be applied to generate a 3D spatial mapping of the space 200 of the venue 100. In some implementations, data from one or more sensor devices 202 are processed to generate a 3D spatial mapping. In various implementations, the sensor device 202 may include any of the following: an image capture device (e.g., a camera), a depth sensing camera, an ultrasonic sensor, an IR camera, etc. The 3D spatial mapping of the venue space 200 enables the 3D coordinates of each seat 114 in the venue 100 to be determined. In some implementations, each seat in the venue has a unique assigned identifier. As shown at reference numeral 212, for each seat, the corresponding 3D coordinates may be mapped to the identifier of the seat. In some implementations, in order to facilitate the generation of the 3D spatial mapping, various localization / positioning techniques may be employed, including but not limited to, for example, Wi-Fi-based positioning, magnetic positioning, visual markings, and / or visual recognition of visual features. Sensors employing such positioning techniques may be placed at the seat position to determine the 3D position of the seat in space.

[0061] In some implementations, the 3D spatial mapping also includes the 3D position and orientation of the camera 116 in the 3D space 200 of the venue 100. Using this information, video processing parameters are then generated for each seat at reference numeral 214. For a given seat, the video processing parameters define which cameras' video feeds are to be stitched and how they are stitched based on the viewing direction in the 3D space in order to generate a view of the venue space from the perspective of the given seat (or more precisely, from the perspective corresponding to the 3D coordinates of the given seat). For cameras with variable orientation (e.g., controlled by an operator), if associated with a given seat, the video processing parameters may identify such cameras as potential video feed sources based on the orientation of the cameras, and further define how the videos from such cameras are to be stitched (based on the viewing direction of the virtual reality viewer), such as parameters related to spatial projection, alignment, blending, anchor points, etc. of the video feeds. It should be understood that in various implementations, the video stitching process may be configured to generate 3D or 2D video. Generating 3D video may require generating separate videos for each eye.

[0062] As shown at reference numeral 216, according to an implementation of the present disclosure, audio processing parameters are generated for each seat in venue 100. In some implementations, 3D space is acoustically modeled using 3D space mapping to determine the acoustic characteristics of the 3D space. In some implementations, microphones are placed at various locations in the venue to determine the acoustic characteristics. For example, sounds can be played through the speakers 118 of the venue, and the audio recorded from the microphones can be analyzed based on the known positions of the microphones to determine the acoustic characteristics and implement acoustic modeling of the 3D space. This can be used to generate audio processing parameters for each seat. The audio processing parameters for a given seat can define how to process audio from various sources (such as audio during game play, commentator audio, audio from live microphones in venue 100, etc.) to provide audio to virtual reality viewers assigned to a given seat.

[0063] As shown at reference numeral 218, the video processing parameters and audio processing parameters for a given seat are stored in association with an identifier (ID) of the seat. In some implementations, the video processing parameters and audio processing parameters for a given seat form at least a portion of a seat profile for the given seat in the venue.

[0064] Although the foregoing mapping of the venue space has been described with reference to seats in the venue, it should be understood that these concepts can be applied to any designated location within the venue. In some implementations, one or more designated locations within the venue (which may or may not coincide with specific seats in the venue) are identified, and based on the above, video processing parameters and audio processing parameters are determined for the designated locations, for example, to provide a 360-degree view from the perspective of the designated locations. In some implementations, the virtual reality viewer can select from such available locations to view the live event. That is, the virtual reality viewer can select a view or a group of views to stream the live event for viewing through his or her HMD.

[0065] Figure 3 The field of view of a virtual reality viewer according to an implementation of the present disclosure is conceptually illustrated. As shown, the virtual reality viewer 120 has a 360-degree field of view 300 conceptually represented in the illustrated implementation. The 360-degree field of view 300 can be, but is not limited to, a horizontal field of view in some implementations, or a vertical field of view in some implementations, for example, or represents any other plane of view in various implementations.

[0066] As described above, the view of the e-sports venue 100 provided to the virtual reality spectator 120 may be stitched together from video feeds from multiple cameras in the e-sports venue 100. In the illustrated implementation, camera 304a provides a video feed with field of view 306a; camera 304b provides a video feed with field of view 306b; camera 304c provides a video feed with field of view 306c; and camera 304d provides a video feed with field of view 306d. Depending on the direction in which the virtual reality spectator 120 is looking, various video feeds from the video feeds of the different cameras are selected to be stitched together to form a video to be rendered to the HMD (150) that provides the appropriate view to the virtual reality spectator 120. In the illustrated implementation, the virtual reality spectator 120 is shown looking in a certain direction so as to have field of view 302. Based on this, the video rendered for viewing by the virtual reality spectator 120 may be stitched together from the video feeds of cameras 304a, 304b, and 304d. The video feed from camera 304 c is not used to provide the current view because camera 304 c does not cover any portion of the area encompassed by the virtual reality viewer's field of view 302 .

[0067] It should be understood that the various cameras have different positions in the 3D venue space and therefore have different viewing angles. Therefore, the video feeds from these cameras are used or not used to provide a view to a particular virtual reality viewer (and the stitching method, if used) can further depend on the directionality and / or viewing angle position of the camera. For example, in some implementations, a video feed that covers the area currently being viewed by the virtual reality viewer but is in a direction substantially opposite to the viewing direction of the virtual reality viewer is not used to provide video to the virtual reality viewer.

[0068] As already mentioned, some cameras may have a fixed orientation, while other cameras may have a variable orientation that may change. Therefore, in some implementations, the current orientation information of the camera is obtained and used to determine whether to use the video feed of a given camera, and / or how to use the video feed of a given camera if selected.

[0069] In some implementations, the orientation of a given camera is adjusted based on the field of view of the virtual reality viewer. For example, the orientation of the camera can be adjusted so that its field of view covers at least a portion of the area being viewed by the virtual reality viewer. In some implementations, the orientation of a given camera is adjusted based on the fields of view of multiple virtual reality viewers in a manner that optimizes the field of view of the given camera. In some implementations, a density map of the fields of view of the virtual reality viewer is determined, which identifies and / or quantifies the relative amount of various spatial regions being viewed. Based on this information, the orientation of the camera can be determined and adjusted, such as to prioritize spatial regions that are being viewed to a higher degree.

[0070] It should be understood that the available video feeds from the various cameras may not provide coverage of all areas that a given virtual reality viewer is viewing (including the necessary directions and / or viewing angles). Areas of the venue space that may be available for live viewing may be defined based on the coverage areas provided by the available video feeds. Figure 3 , the fields of view 306a, 306b, 306c, and 306d of the available cameras together define a live view area 308, which is an area of ​​the 360 ​​degree field of view 300 in which a live, real-time view of the venue space can be provided, as determined by the venue space coverage provided by the video feeds from the available cameras.

[0071] In some implementations, the pre-recorded view may be presented to the remaining area not provided with a live view, thereby defining a pre-recorded view area 310 in the illustrated implementation. That is, a video or image of the area 310 may be recorded from an earlier time (e.g., when one or more cameras were oriented so as to cover at least a portion of the area 310), and the pre-recorded video / image may be stitched and rendered to provide a view of the area 310. It should be appreciated that in some implementations, when the field of view of the virtual reality viewer 120 encompasses portions of both the live view area 308 and the pre-recorded view area 310, the view presented to the virtual reality viewer 120 is a composite image generated from both the live video and the pre-recorded video / image.

[0072] In other implementations, other content, such as advertising / sponsor content, game-related content, etc., may be presented to areas of the virtual reality viewer's field of view 300 that are not available for live viewing.

[0073] Continue to refer Figure 3 As indicated at reference numeral 320, the video feeds from the different cameras may be stitched together to form a video based on the current field of view 302 of the virtual reality viewer 120, thereby providing an appropriate view for the virtual reality viewer 120. In addition, as indicated at reference numeral 322, the stitched video may undergo further processing, such as a compression process, to reduce the amount of data transmitted. In some implementations, the compression process may include foveated rendering, whereby areas of an image frame that the virtual reality viewer 120 is determined to look at (e.g., as determined based on gaze direction information detected by the HMD 150) are rendered to have a higher image quality than other areas (such as by using increased resolution, update frequency, bit rate, color, dynamic range, sharpness, contrast, or any other parameter that affects image quality). The output of the compression process is an image frame 324, which defines a video for the virtual reality viewer 120. As shown in the illustrated implementation, area 326 in the video image frame is rendered to have a higher image quality than area 328.

[0074] The compressed image frames are then transmitted / streamed through the virtual reality viewer's HMD 150 for viewing, as shown at reference numeral 330 .

[0075] Figure 4 A system for providing virtual reality viewing of e-sports events according to an implementation of the present disclosure is conceptually shown. Although not described in detail for ease of description, it should be understood that the various systems, components, and modules described herein can be defined by one or more computers or servers having one or more processors for executing program instructions and one or more memory devices for storing data and the program instructions. It should be understood that any one of such systems, components, and modules can communicate with any other of such systems, components, and modules through one or more networks, and / or transmit / receive data, as needed, to facilitate the functionality of the implementation of the present disclosure. In various implementations, the various parts of the systems, components, and modules can be local to each other or distributed on one or more networks.

[0076] In the illustrated implementation, the virtual reality viewer 120 interfaces with the system through an HMD 150 and performs additional interactions and inputs using one or more controller devices 152. In some implementations, the video images displayed to the virtual reality viewer 120 via the HMD are received from a computing device 400 that communicates with various systems and devices over a network 402 (which may include the Internet), as described herein.

[0077] To initiate access to watch an e-sports event, the virtual reality spectator 120 may access an event manager 404, which processes requests to watch an e-sports event. The event manager 404 may include seat assignment logic 405, which is configured to assign the virtual reality spectator 120 to a specific seat in a venue for an e-sports event. The seat assignment logic 405 may utilize various types of information to determine which seat to assign the virtual reality spectator 120 to, including based on user profile data 408 of the spectator stored in a user database 407. For example, such user profile data 408 may include: demographic information about the user, such as age, geographic location, gender, nationality, primary language, occupation, etc.; and other types of information, such as interests, preferences, games played / owned / purchased, gaming experience level, Internet browsing history, etc.

[0078] In some implementations, the seat assignment logic 405 may also use information obtained from the social network 410 to, for example, assign audience members who are friends on the social network to seats that are close to or adjacent to each other. To obtain such information, the social network 410 may store social information about the user (including social graph membership information) as social data 414 to the social database 412. In some implementations, the seat assignment logic 405 may access the social data (e.g., access the social graph of a given user / audience member) through an API of the social network 410.

[0079] In some implementations, the seat assignment logic 405 is configured to determine which seats are available, e.g., not occupied by real and / or virtual spectators, and assign virtual reality spectators based at least in part on such information. In some implementations, the seat assignment logic 405 is configured to automatically assign virtual reality spectators to the best available seats, as determined according to a predefined ranking of seats in the venue.

[0080] It should be understood that the seat assignment logic 405 can use any of the factors described herein in combination with any other factors to determine which seat to assign a given virtual reality viewer to. In some implementations, the available seats are scored based on various factors, and the seat assignment is determined based on the scores (e.g., assigning the virtual reality viewer to the seat with the highest score). In some implementations, the seat assignment logic 405 presents a recommended seat for the viewer 120 to accept, and when the viewer 120 accepts the recommended seat, the viewer 120 is assigned to the recommended seat.

[0081] In other implementations, the virtual reality viewer 120 may access an interface provided by the seat selection logic 406 that is configured to enable the virtual reality viewer to select a given seat from the available seats.

[0082] The venue database 416 stores data about one or more venues as venue data 418. The venue data 418 may include any data describing the venue, such as a 3D spatial map; the location of cameras, speakers, microphones, etc. The venue data 418 may also include a table 420 that associates seat profiles to unique seat identifiers. In some implementations, each seat has its own seat profile. In some implementations, a group of seats (e.g., immediately adjacent to each other) may share the same seat profile. An exemplary seat profile 422 includes information such as a 3D position 424 of a seat, video processing parameters 426, and audio processing parameters 428.

[0083] The video processor 432 includes a stitching processor 434 that can stitch together the video feeds 430 from the camera 116 using the video processing parameters 426 and / or the 3D position 424 of the spectator's assigned seat to generate a composite video that provides a view for the virtual reality spectator 120 according to the viewing direction of the virtual reality spectator. In some implementations, a spatial modeling module 438 generates or accesses a spatial model of the 3D environment of the venue (e.g., including the positions of the cameras and the positions of the spectator's seats) to facilitate the stitching of the video feeds 430. The stitching of the video feeds may require spatial projection of the video feeds to provide the spectator with a perspectively correct video. In some implementations, the resulting composite video is a 3D video, while in other implementations, the composite video is a 2D video.

[0084] The compression processor 436 is configured to compress the original composite video using video compression techniques known in the art and foveated rendering to reduce the amount of data required for streaming. The streaming server 448 then streams the compressed video data to the computing device 400 via the network 402, and the computing device 400 processes the video and / or renders the video to the HMD 150 for viewing by the virtual reality viewer 120.

[0085] In some implementations, the video feed is transmitted from the camera to one or more computing devices local to the camera / venue, which also performs video processing. In some implementations, the camera is directly connected to such a computing device. In some implementations, the video feed is transmitted to such a computing device over a local network (e.g., including a local area network (LAN), a Wi-Fi network, etc.). In some implementations, the computing device is located remotely, and the video feed may be transmitted over one or more networks (such as the Internet, a LAN, a wide area network (WAN), etc.).

[0086] The audio processor 444 is configured to process the audio data 442 from the audio source 440 for streaming with the compressed video data. The processing may use the audio processing parameters 428 and / or the 3D position 424 of the audience seat. In some implementations, the audio modeling module 446 applies an audio model based on the 3D space of the venue to process the audio data 442. Such an audio model may simulate the acoustics of the assigned seats in the venue so that the audio is presented to the virtual reality audience in a realistic manner. For example, but not limited to, the sound from other virtual reality audiences may be processed to simulate not only the directionality of the seat position relative to the virtual reality audience 120, but also the appropriate acoustics (e.g., delay, reverberation, etc.) of the seat position in the venue. As described, the audio source may include audio of the game play process, commentators, house music, audio from microphones in the venue, etc.

[0087] Figure 5 A technique for determining whether a seat in a venue is occupied by a real audience member according to an implementation of the present disclosure is shown. In the implementation shown, various seats 500, 502, and 504 in the venue 100 are shown. Seat 502 is occupied by a real audience member 506. Camera 512 can be configured to capture an image of the seat. The captured image can be analyzed according to an image recognition process 514, which can search for an image of an indicator that a given seat is occupied. For example, but not limitation, the image recognition process can include: facial recognition or other forms of people recognition for identifying the presence of a person occupying a given seat, motion detection for detecting motion occurring near a given seat, image recognition of empty seat configurations, etc.

[0088] In some implementations, the seat may include a tag (or have a tag fixed thereon) that may be identified by the image recognition process 514. In the illustrated implementation, tag 508 is attached to seat 500, and tag 510 is attached to seat 504. Seat 502 is occupied by real audience 506, so the tag attached to seat 502 is no longer visible and will not be identified. In this way, an empty seat can be identified by identifying the tag in the image captured by camera 512. Tags may take any recognizable form in various implementations, including but not limited to, for example, reflectors or reflective materials, recognizable images / patterns / graphics, lights (e.g., LEDs), specific colors, etc.

[0089] The tag may be attached to any portion of the seat along the following, including but not limited to, for example, along the seat back or seat cushion in some implementations, more specifically, along the top of the seat back or seat cushion; along the front of the seat cushion, if the seat cushion is folded, so that the tag is visible when the seat cushion is folded when the seat is not occupied; along the armrests, etc. In addition, it should be understood that a given seat may have any number of tags attached.

[0090] As shown at reference numeral 516, based on the detection of an empty seat or an occupied seat according to the above, the occupancy status of a given seat can be updated to reflect whether it is occupied or vacant. As previously mentioned, in some implementations, an empty seat can be used for virtual reality viewing of venue 100.

[0091] Figure 6A method is shown for a virtual reality spectator to see himself / herself in the context of a real venue according to an implementation of the present disclosure. In the implementation shown, an e-sports event takes place in a venue 100 in which there are many seats. For example, seat 136 is occupied by a real spectator 138. Virtual reality spectator 120 is able to "occupy" seat 122 and thus experience the view of venue 100 through HMD 150 as if virtual reality spectator 120 were actually present in seat 122.

[0092] Camera 600 may be operated by operator 602 and oriented toward seats 122 and 136, thereby capturing video of real audience 138 in seats 136. This video, or a portion thereof, may be projected onto display 108 c, which may be one of a large display that may be viewed simultaneously by many audience members in venue 100. As shown, the projected video rendered on display 108 c shows image 606 of real audience 138.

[0093] However, in some implementations, when the virtual reality viewer 120 looks at the display 108 c, he / she will also see himself / herself in the projected video shown on the display 108 c. That is, when the video projected on the display 108 c includes the location of the seat 122 that the virtual reality viewer "occupies" (or is assigned to), then in the view provided to the virtual reality viewer 120 by the HMD that includes the display 108 c, the rendered image of the display 108 c is changed to an image that includes the virtual reality viewer 120 (or the avatar 604 associated with the virtual reality viewer 120) in the appropriate position so as to occupy the seat 122. In this way, the virtual reality viewer 120 is able to see himself / herself in the seat 122 of the venue 100, thereby further increasing the realism and interactivity of the experience of the virtual reality viewer 120. As shown in the illustrated implementation, the rendered video on the display 108 c viewed by the virtual reality viewer 120 will include the image 608 of the virtual reality viewer 120 (or its associated avatar 604).

[0094] It should be understood that in some implementations, the video actually rendered on the display 108 c in the venue 100 is not changed to include the image of the virtual reality viewer. However, in other implementations, the video actually rendered on the display 108 c in the venue 100 is changed to include the image of the virtual reality viewer.

[0095] Continue to refer Figure 6, a method for performing the above-described video changes to include a virtual reality viewer is shown. At operation 610, a video is captured by a camera in the venue, the video including a position / seat assigned to a virtual reality viewer, the position / seat defining a position from which a view of the venue is provided to the virtual reality viewer through an HMD.

[0096] At operation 612, the seats and / or positions captured by the camera are determined. This may require determining the 3D position and / or orientation of the camera within the venue in order to determine which seats are being captured in the video. For example, using the known position and orientation of the camera, the 3D space within the venue captured in the video can be determined, and using a mapping of the seats to the 3D positions in the venue, the specific seats captured in the video can be identified. For example, but not limited to, the position / orientation of the camera can be determined using any type of positioning technology known in the art, including Wi-Fi-based positioning, magnetic positioning, recognition of visual markers and / or visual features, etc.

[0097] At operation 614, for a given identified seat, it is determined whether a virtual reality viewer is assigned to the seat, and if so, the virtual reality viewer is identified. At operation 616, when the view provided to the virtual reality viewer includes a display in the venue on which the captured video is to be rendered, the view of the display is modified so as to show the virtual reality viewer or its representative avatar in the seat occupied by the virtual reality viewer. For example, the viewing direction of the virtual reality viewer can be tracked to determine when the view includes the display. When the view includes the display, the view can be modified as discussed above to include the virtual reality viewer. In some implementations, an image of the virtual reality viewer (or its avatar) is formed as an overlay in the display view presented to the virtual reality viewer. In some implementations, a camera video feed including the display is modified to include an image of the virtual reality viewer (or avatar), and as previously discussed, the modified video feed is stitched to form a view for the virtual reality viewer.

[0098] Figure 7 Seats in a venue with additional functionality for sensing real spectators and enabling interactivity for virtual reality spectators are shown in accordance with implementations of the present disclosure. In the implementation shown, seat 700 and seat 702 are shown, with seat 702 being in front of seat 700. Sensor units 704 and 706 are also shown, which are configured for seats 700 and 702, respectively. In some implementations, the sensor units are separate attachments that are removable from the seats. In some implementations, the sensor units and / or any of their components as described herein may be incorporated as part of the seats.

[0099] For ease of description, sensor unit 706 is described herein, and it should be understood that this also applies to sensor unit 704. Sensor unit 706 includes one or more occupancy sensors 708, which are configured to detect whether seat 702 is occupied by a real audience. As shown, occupancy sensor 708 can be positioned along the bottom of the seat and / or the seat back. In some implementations, occupancy sensor 708 includes one or more pressure sensors, which detect when pressure is applied to sense whether a person is sitting in seat 702. In addition, such a pressure sensor can determine / quantify how much pressure (force) is applied, which can make it possible to distinguish the weight of a person from another object. In some implementations, occupancy sensor 708 includes one or more proximity sensors, which can detect when another object is close (e.g., within a predefined distance) and may detect the distance of the object from the proximity sensor.

[0100] In some implementations, the sensor unit 706 also includes a camera 710 configured to capture video / images from the perspective of the seat 702, which can be used to provide a view of the venue to the virtual reality spectator assigned to the seat 702. In some implementations, the camera 710 is a spherical panoramic camera capable of capturing a 360-degree field of view. In various implementations, the camera 710 includes one or more image capture devices. In some implementations, the camera 710 is a stereo camera capable of providing stereo images with 3D depth.

[0101] In some implementations, the sensor unit 706 also includes one or more microphone arrays 716, each microphone array including a plurality of microphones. Sound from the microphone arrays may be processed to identify the location of a sound source based on analyzing differences in sound reception timing between different microphones. Sound captured by the microphone arrays may be processed and / or provided to a virtual reality spectator assigned to the seat 702, so that the virtual reality spectator can experience sound from the location in the venue where the spectator is virtually located.

[0102] In some implementations, the sensor unit 712 includes hardware that enables the virtual reality audience to interact with the audience in the adjacent or nearby seats. In the illustrated implementation, the sensor unit 712 also includes a display 712 and a speaker 714. The display 712 can display an image of the virtual reality audience (or avatar) assigned to the seat 702 to allow the real audience in the nearby seats to see the virtual reality audience (or their avatar). In some implementations, the virtual reality audience can communicate through the speaker 714 so that the sound from the virtual reality audience (e.g., captured by the microphone of the virtual reality audience's HMD or in its local environment) can be transmitted to the sensor unit 706 and output through the speaker 714. Similarly, the real audience nearby can speak, and this will be captured by the microphone array 716 and transmitted to the virtual reality audience's HMD, so that the virtual reality audience can hear the voice of the real audience nearby. In this way, the real audience and the virtual reality audience can interact with each other. It should be understood that in some implementations, the virtual reality audience can see the real audience nearby through the camera 710.

[0103] In some implementations, the sensor unit 706 includes a rear-facing sensor 718 that can be configured to detect occupancy of the seat 700 behind the seat 702. In some implementations, the rear-facing sensor can be a proximity sensor and can use IR, ultrasonic, or other sensing technology.

[0104] In some implementations, the sensor unit 706 includes positioning technology that enables the location of the sensor unit to be determined. For example, but not by way of limitation, this may include Wi-Fi positioning, magnetic positioning, GPS, etc.

[0105] The sensor unit 706 may further communicate and transmit / receive data over one or more networks, such as cellular, Wi-Fi, or other types of networks, including wireless and / or wired networks.

[0106] In some implementations, the sensor unit 706 includes a tag that is identifiable through image recognition processing so that the sensor unit can be identified in images captured by cameras in the venue.

[0107] Although the implementation of the present disclosure has been generally described with reference to e-sports events and e-sports venues, it should be understood that in other implementations, the principles of the present disclosure can be applied to any other type of live events for viewing and corresponding venues, including but not limited to, for example, sporting events (e.g., football, basketball, baseball, soccer, hockey, etc.), concerts, ceremonies, speeches, etc.

[0108] Figure 8A system for interacting with a virtual environment via a head mounted display (HMD) according to an implementation of the present disclosure is shown. The HMD may also be referred to as a virtual reality (VR) headset. As used herein, the term "virtual reality" (VR) generally refers to the interaction of a user with a virtual space / environment, which involves viewing the virtual space through the HMD (or VR headset) in a manner that responds in real time to the movement of the HDM (controlled by the user) to provide the user with a feeling of being in the virtual space. For example, when facing a given direction, the user can see a three-dimensional (3D) view of the virtual space, and when the user turns to one side and thus also turns the HMD, the view of that side in the virtual space is rendered on the HMD. In the implementation shown, a user 120 wearing a head mounted display (HMD) 150 is shown. The HMD 150 is worn in a manner similar to glasses, goggles, or a helmet, and is configured to display video games or other content to the user 120. The HMD 150 provides a very immersive experience by virtue of its ability to bring the display mechanism very close to the user's eyes. Thus, HMD 150 can provide a display area to each eye of the user that occupies most or even the entire field of view of the user, and can also provide viewing with three-dimensional depth and perspective.

[0109] In the implementation shown, the HMD 150 is wirelessly connected to the computer 806. In other implementations, the HMD 150 is connected to the computer 806 via a wired connection. The computer 806 may be any general or special purpose computer known in the art, including but not limited to a game console, a personal computer, a laptop computer, a tablet computer, a mobile device, a cellular phone, a tablet, a thin client, a set-top box, a media streaming device, etc. In some implementations, the computer 106 may be configured to execute a video game, and output video and audio from the video game to be rendered by the HMD 150. In some implementations, the computer 806 is configured to execute any other type of interactive application that provides a virtual space / environment that can be viewed through the HMD. The transceiver 810 is configured to transmit (via a wired connection or a wireless connection) the video and audio from the video game to the HMD 150 for presentation on the HMD 150. The transceiver 810 includes a transmitter for transmitting data to the HMD 150, and a receiver for receiving data transmitted by the HMD 150.

[0110] In some implementations, the HMD 150 may also communicate with the computer through alternative mechanisms or channels, such as via a network 812 connecting both the HMD 150 and the computer 106 .

[0111] The user 120 may manipulate the interface objects 152 to provide input for the video game. In addition, the camera 808 may be configured to capture images of the interactive environment in which the user 120 is located. These captured images may be analyzed to determine the position and movement of the user 120, the HMD 150, and the interface objects 152. In various implementations, the interface objects 152 include optical and / or inertial sensors that may be tracked to enable determination of the position and orientation of the interface objects and tracking of movement.

[0112] In some implementations, a magnetic source 816 is provided that emits a magnetic field to enable magnetic tracking of the HMD 150 and interface object 152. Magnetic sensors in the HMD 150 and interface object 152 can be configured to detect magnetic fields (e.g., strength, orientation), and this information can be used to determine and track the position and / or orientation of the HMD 150 and interface object 152.

[0113] In some implementations, the interface object 152 is tracked relative to the HMD 150. For example, the HMD 150 may include an outward-facing camera that captures images that include the interface object 152. The captured images may be analyzed to determine the position / orientation of the interface object 152 relative to the HMD 150, and using the known position / orientation of the HMD, the position / orientation of the interface object 152 in the local environment.

[0114] The manner in which a user interacts with a virtual reality scene displayed in the HMD 150 may vary, and other interface devices besides the interface objects 152 may be used. For example, various single-handed controllers as well as two-handed controllers may be used. In some implementations, the controllers themselves may be tracked by tracking lights included in the controllers or by tracking shapes, sensors, and inertial data associated with the controllers. Using these different types of controllers or even just gestures made and captured by one or more cameras, it is possible to interface with, control, manipulate, interact with, and participate in the virtual reality environment presented on the HMD 150.

[0115] In addition, the HMD 150 may include one or more lights that can be tracked to determine the position and orientation of the HMD 150. The camera 808 may include one or more microphones to capture sounds from the interactive environment. The sounds captured by the microphone array may be processed to identify the location of the sound source. The sounds from the identified location may be selectively utilized or processed to exclude other sounds that are not from the identified location. In addition, the camera 808 may be defined to include multiple image capture devices (e.g., a pair of stereo cameras), IR cameras, depth cameras, and combinations thereof.

[0116] In some implementations, the computer 806 is used as a thin client that communicates with a cloud application (e.g., gaming, streaming, viewing, etc.) provider 814 via a network 812. In such an implementation, generally speaking, the cloud application provider 114 maintains and executes the video game being played by the user 150. The computer 806 transmits input from the HMD 150, the interface object 152, and the camera 808 to the cloud application provider, which processes the input to affect the game state of the video game being executed. Output from the video game being executed (such as video data, audio data, and tactile feedback data) is transmitted to the computer 806. The computer 806 can further process the data before transmission, or the data can be transmitted directly to the relevant device. For example, the video and audio streams are provided to the HMD 150, and the tactile / vibration feedback commands are provided to the interface object 152.

[0117] In some implementations, the HMD 150, interface object 152, and camera 808 themselves may be networked devices connected to a network 812 to communicate, for example, with a cloud application provider 814. In some implementations, the computer 106 may be a local network device, such as a router, that does not otherwise perform video game processing but facilitates the passage of network traffic. The connection of the HMD 150, interface object 152, and camera 108 to the network may be wired or wireless.

[0118] In addition, although implementations in the present disclosure may be described with reference to a head-mounted display, it should be understood that in other implementations, a non-head-mounted display may be used instead, including but not limited to: a portable device screen (e.g., a tablet, smart phone, laptop computer, etc.) or any other type of display that can be configured to render video and / or provide a display of an interactive scene or virtual environment according to the present implementation.

[0119] Figure 9A-1 and Figure 9A-2 A head mounted display (HMD) according to an implementation of the present disclosure is shown. Specifically, Figure 9A-1 Shows VR headset, which is an example of an HMD according to an implementation of the present disclosure. As shown, HMD 150 includes a plurality of lights 900A-900H. Each of these lights can be configured to have a specific shape and can be configured to have the same or different colors. Lights 900A, 900B, 900C, and 900D are arranged on the front surface of HMD 150. Lights 900E and 900F are arranged on the side surfaces of HMD 150. And lights 900G and 900H are arranged at the corners of HMD 150 so as to span the front and side surfaces of HMD 150. It should be understood that the lights can be identified in the captured images of the interactive environment in which the user uses HMD 150. Based on the identification and tracking of the lights, the position and orientation of HMD 150 in the interactive environment can be determined. It should also be understood that some lights may or may not be visible depending on the specific orientation of HMD 150 relative to the image capture device. Furthermore, depending on the orientation of the HMD 150 relative to the image capture device, different portions of the lights (eg, lights 900G and 900H) may be exposed for image capture.

[0120] In one implementation, the lights may be configured to indicate the current state of the HMD to others nearby. For example, some or all of the lights may be configured to have a specific color arrangement, intensity arrangement, configured to flash, have a specific on / off configuration, or other arrangements indicating the current state of the HMD 150. For example, the lights may be configured to display a different configuration from other non-active game play aspects of the video game during an active game play process of the video game (usually during an active timeline or during a game play process occurring within a game scene), such as a navigation menu interface or configuration game settings (during which the game timeline or scene may not be active or paused). The lights may also be configured to indicate the relative intensity level of the game play process. For example, when the intensity of the game play process increases, the light intensity or flashing rate may increase. In this way, people outside the user can view the lights on the HMD 150 and understand that the user is actively participating in an intense game play process and may not want to be disturbed at that time.

[0121] The HMD 150 may additionally include one or more microphones. In the illustrated implementation, the HMD 150 includes microphones 904A and 904B defined on the front surface of the HMD 150 and a microphone 904C defined on the side surface of the HMD 150. By using an array of microphones, the sound from each microphone may be processed to determine the location of the sound source. This information may be used in a variety of ways, including excluding unwanted sound sources, associating sound sources with visual identification, and the like.

[0122] The HMD 150 may also include one or more image capture devices. In the illustrated implementation, the HMD 150 is shown to include image capture devices 902A and 902B. By utilizing a pair of stereoscopic image capture devices, three-dimensional (3D) images and videos of the environment can be captured from the perspective of the HMD 150. Such videos can be presented to the user to provide the user with a "video viewing" capability when wearing the HMD 150. That is, although the user cannot see through the HMD 150 in a strict sense, the video captured by the image capture devices 902A and 902B (e.g., or one or more outward-facing (e.g., forward-facing) cameras disposed outside the HMD 150 body) can still provide a functional equivalent of being able to see the environment outside the HMD 150 (just as if viewed through the HMD 150). Such videos can be enhanced with virtual elements to provide an augmented reality experience, or can be combined or mixed with virtual elements in other ways. Although in the illustrated implementation, two cameras are shown on the front surface of the HMD 150, it should be understood that any number of outward-facing cameras oriented in any direction may be mounted on the HMD 150. For example, in another implementation, there may be cameras mounted on the sides of the HMD 150 to provide additional panoramic image capture of the environment. Additionally, in some implementations, such outward-facing cameras may be used to track other peripheral devices (e.g., controllers, etc.). That is, the position / orientation of the peripheral device relative to the HMD may be identified and tracked in the captured images from such outward-facing cameras on the HMD, and using the known position / orientation of the HMD in the local environment, the true position / orientation of the peripheral device may then be determined.

[0123] Fig. 9B An example is shown where a user 120 of an HMD 150 interfaces with a client system 806 and the client system 806 provides content to a second screen display, which is referred to as second screen 907. The client system 806 may include integrated electronics for handling content sharing from the HMD 150 to the second screen 907. Other implementations may include separate devices, modules, connectors that will interface between the client system and each of the HMD 150 and the second screen 907. In this general example, the user 120 is wearing the HMD 150 and is playing a video game using a controller (which may also be an interface object 104). The interactive game that the user 120 is playing will generate video game content (VGC), which is interactively displayed to the HMD 150.

[0124] In one implementation, the content displayed in the HMD 150 is shared to the second screen 907. In one example, a person viewing the second screen 907 can view the content interactively played by the user 120 in the HMD 150. In another implementation, another user (e.g., player 2) can interact with the client system 806 to generate second screen content (SSC). The second screen content generated by the player who also interacts with the controller 104 (or any type of user interface, gesture, voice, or input) can be generated to the client system 806 as an SSC, which can be displayed on the second screen 907 together with the VGC received from the HMD 150.

[0125] Thus, interactivity with other users who may be co-located with the HMD user or who may be remote from the HMD user can be social, interactive, and more immersive for both the HMD user and users who may be viewing content played by the HMD user on a second screen 907. As shown, the client system 806 can be connected to the Internet 910. The Internet can also provide access to the client system 806 to obtain content from various content sources 920. The content sources 920 can include any type of content that can be accessed through the Internet.

[0126] Such content may include, but is not limited to, video content, movie content, streaming content, social media content, news content, friend content, advertising content, etc. In one implementation, the client system 806 may be used to simultaneously process content for the HMD user so that multimedia content associated with interactivity during the game play process is provided to the HMD. The client system 806 may then also provide other content that may not be related to the video game content to the second screen. In one implementation, the client system 806 may receive the second screen content from one of the content sources 920 or from a local user or a remote user.

[0127] Fig.10 The functionality of the HMD 150 according to implementations of the present disclosure is conceptually illustrated in conjunction with an executing video game or other application. The executing video game / application is defined by a game / application engine 1020 that receives input to update the game / application state of the video game / application. The game state of the video game may be defined, at least in part, by various parameter values ​​of the video game that define various aspects of the current gameplay (such as the presence and location of objects, conditions of the virtual environment, triggering of events, user profiles, viewing perspectives, etc.).

[0128] In the illustrated implementation, for example, the game engine receives controller input 1014, audio input 1016, and motion input 1018. The controller input 1014 may be defined in terms of the operation of a game controller separate from the HMD 150, such as a handheld game controller (e.g., a Sony 4 Wireless Controller, Sony Move motion controller) or interface object 152. For example, controller input 1014 may include directional input, button pressing, trigger activation, movement, gesture or other types of input processed according to the operation of the game controller. In some implementations, the movement of the game controller is tracked by an outward-facing camera 1011 of the HMD 102, which provides the position / orientation of the game controller relative to the HMD 102. Audio input 1016 can be processed from a microphone 1002 of the HMD 150 or from a microphone included in the image capture device 808 or elsewhere in the local environment. Motion input 1018 can be processed from a motion sensor 1000 included in the HMD 150, or from an image capture device 808 when the image capture device 808 captures an image of the HMD 150. The game engine 1020 receives input processed according to the configuration of the game engine to update the game state of the video game. The game engine 1020 outputs the game state data to various rendering modules, which process the game state data to define the content to be presented to the user.

[0129] In the illustrated implementation, a video rendering module 1022 is defined to render a video stream for presentation on the HMD 150. The video stream may be presented by the display / projector mechanism 1010 and viewed by the user's eyes 1006 through the optics 1008. An audio rendering module 1004 is configured to render an audio stream for the user to hear. In one implementation, the audio stream is output through a speaker 1004 associated with the HMD 150. It should be understood that the speaker 1004 may take the form of an open-air speaker, headphones, or any other type of speaker capable of presenting audio.

[0130] In one implementation, a gaze tracking camera 1012 is included in the HMD 150 to enable tracking of the user's gaze. The gaze tracking camera captures images of the user's eyes, which are analyzed to determine the user's gaze direction. In one implementation, information about the user's gaze direction can be used to influence video rendering. For example, if the user's eyes are determined to be looking in a particular direction, the video rendering for that direction can be prioritized or emphasized, such as by providing more details or faster updates in the area the user is viewing. It should be understood that the user's gaze direction can be defined relative to the head-mounted display, relative to the real environment in which the user is located, and / or relative to a virtual environment being rendered on the head-mounted display.

[0131] Broadly speaking, analysis of the images captured by the gaze tracking camera 1012, when considered alone, provides a gaze direction of the user relative to the HMD 150. However, when considered in conjunction with the tracked position and orientation of the HMD 150, the user's real-world gaze direction can be determined, since the position and orientation of the HMD 150 are synonymous with the position and orientation of the user's head. That is, the user's real-world gaze direction can be determined by tracking the positional movements of the user's eyes and tracking the position and orientation of the HMD 150. When a view of the virtual environment is rendered on the HMD 150, the user's real-world gaze direction can be applied to determine the user's virtual-world gaze direction in the virtual environment.

[0132] In addition, the haptic feedback module 1026 is configured to provide signals to haptic feedback hardware included in the HMD 150 or another device operated by the user, such as the interface object 152. The haptic feedback can take the form of various types of tactile sensations, such as vibration feedback, temperature feedback, pressure feedback, etc. The interface object 152 can include corresponding hardware for rendering such forms of haptic feedback.

[0133] refer to Fig.11 , a diagram showing components of a head mounted display 150 according to an implementation of the present disclosure. The head mounted display 150 includes a processor 1100 for executing program instructions. A memory 1102 is provided for storage purposes, and the memory 1102 may include both volatile memory and non-volatile memory. A display 1104 is included, which provides a visual interface that a user can view. A battery 1106 is provided as a power source for the head mounted display 150. The motion detection module 1108 may include any of a variety of motion sensitive hardware, such as a magnetometer 1110, an accelerometer 1112, and a gyroscope 1114.

[0134] An accelerometer is a device used to measure acceleration and the reaction force caused by gravity. Single-axis and multi-axis models can be used to detect the magnitude and direction of acceleration in different directions. Accelerometers are used to sense tilt, vibration, and shock. In one implementation, three accelerometers 1112 are used to provide the direction of gravity, which gives an absolute reference for two angles: world space pitch and world space roll.

[0135] The magnetometer measures the strength and direction of the magnetic field near the head-mounted display. In one implementation, three magnetometers 1110 are used in the head-mounted display to ensure an absolute reference for the world-space yaw angle. In one implementation, the magnetometer is designed to span the earth's magnetic field, which is ±80 microtesla. The magnetometer is affected by metal and provides a yaw measurement that is monotonic to the actual yaw. The magnetic field may be distorted by metal in the environment, which causes distortion in the yaw measurement. If necessary, information from other sensors (such as a gyroscope or a camera) can be used to calibrate the distortion. In one implementation, an accelerometer 1112 is used together with the magnetometer 1110 to obtain the inclination and azimuth of the head-mounted display 150.

[0136] In some implementations, the magnetometer of the head mounted display is configured to be read when electromagnets in other nearby devices are inactive.

[0137] A gyroscope is a device for measuring or maintaining orientation based on the principle of angular momentum. In one implementation, three gyroscopes 1114 provide information about movement across corresponding axes (x, y, and z) based on inertial sensing. Gyroscopes help detect rapid rotations. However, gyroscopes can drift over time in the absence of an absolute reference. This requires periodically resetting the gyroscopes, which can be done using other available information such as position / orientation determination based on visual tracking of the object, accelerometers, magnetometers, etc.

[0138] Camera 1116 is provided for capturing images and image streams of the real environment. More than one camera may be included in the head mounted display 150, including a rear-facing camera (pointed away from the user when the user is viewing the display of the head mounted display 150), and a front-facing camera (pointed toward the user when the user is viewing the display of the head mounted display 150). In addition, a depth camera 1118 may be included in the head mounted display 150 for sensing depth information of an object in the real environment.

[0139] The head mounted display 150 includes a speaker 1120 for providing audio output. In addition, a microphone 1122 may be included for capturing audio from a real environment, including sounds from the surrounding environment, voices emitted by the user, etc. The head mounted display 150 includes a tactile feedback module 1124 for providing tactile feedback to the user. In one implementation, the tactile feedback module 1124 can cause movement and / or vibration of the head mounted display 150 to provide tactile feedback to the user.

[0140] LEDs 1126 are provided as visual indicators of the status of the head mounted display 150. For example, the LEDs may indicate battery charge, power on, etc. A card reader 1128 is provided to enable the head mounted display 150 to read information from and write information to a memory card. A USB interface 1130 is included as an example of an interface for enabling connection of peripheral devices or connection with other devices such as other portable devices, computers, etc. In various implementations of the head mounted display 150, any of various types of interfaces may be included to enable greater connectivity of the head mounted display 150.

[0141] A Wi-Fi module 1132 is included for connecting to the Internet or a local area network via wireless networking technology. In addition, the head mounted display 150 includes a Bluetooth module 1134 for wireless connection with other devices. A communication link 1136 may also be included to connect to other devices. In one implementation, the communication link 1136 uses infrared transmission for wireless communication. In other implementations, the communication link 1136 can use any of a variety of wireless or wired transmission protocols to communicate with other devices.

[0142] An input button / sensor 1138 is included to provide an input interface for the user. Any of a variety of input interfaces may be included, such as buttons, a touch pad, a joystick, a trackball, etc. An ultrasonic communication module 1140 may be included in the head mounted display 150 to facilitate communication with other devices via ultrasonic technology.

[0143] Biosensor 1142 is included to enable detection of physiological data from a user. In one implementation, biosensor 1142 includes one or more dry electrodes for detecting bioelectric signals of a user through the user's skin.

[0144] The video input 1144 is configured to receive a video signal from a host processing computer (e.g., a host game console) for rendering on the HMD. In some implementations, the video input is an HDMI input.

[0145] The foregoing components of the head mounted display 150 have been described as merely exemplary components that may be included in the head mounted display 150. In various implementations of the present disclosure, the head mounted display 150 may or may not include some of the various components described above. Implementations of the head mounted display 150 may additionally include other components not currently described but known in the art to facilitate various aspects of the present disclosure as described herein.

[0146] Fig.12 1 is a block diagram of a gaming system 1200 according to various implementations of the present disclosure. The gaming system 1200 is configured to provide a video stream to one or more clients 1210 via a network 1215. The gaming system 1200 typically includes a video server system 1220 and an optional gaming server 1225. The video server system 1220 is configured to provide a video stream to one or more clients 1210 with a minimum quality of service. For example, the video server system 1220 may receive a game command that changes a state or viewpoint within a video game, and provide an updated video stream reflecting such a state change to the client 1210 with a minimum lag time. The video server system 1220 may be configured to provide a video stream in various alternative video formats, including formats that have not yet been defined. In addition, the video stream may include video frames that are configured to be presented to a user at various frame rates. Typical frame rates are 30 frames per second, 60 frames per second, and 120 frames per second. However, higher or lower frame rates are included in alternative implementations of the present disclosure.

[0147] Client 1210 (referred to as 1210A, 1210B, etc. individually herein) may include a head-mounted display, a terminal, a personal computer, a game console, a tablet computer, a phone, a set-top box, an information kiosk, a wireless device, a digital board, a stand-alone device, a handheld game playing device, and / or a similar device. Typically, client 1210 is configured to receive a coded video stream, decode the video stream, and present the resulting video to a user, such as a player of a game. The process of receiving a coded video stream and / or decoding the video stream typically includes storing a separate video frame in a receiving buffer of the client. The video stream may be presented to the user on a display integrated with client 1210 or on a separate device such as a monitor or a television. Client 1210 is optionally configured to support more than one game player. For example, a game console may be configured to support two, three, four, or more players simultaneously. Each of these players may receive a separate video stream, or a single video stream may include an area of ​​frames specifically generated for each player, for example, an area of ​​frames generated based on the perspective of each player. Client 1210 is optionally geographically dispersed. The number of clients included in the gaming system 1200 can vary widely from one or two to thousands, tens of thousands, or more. As used herein, the term "game player" is used to refer to a person who plays a game, and the term "game playing device" is used to refer to a device for playing a game. In some implementations, a game playing device may refer to multiple computing devices that collaborate to deliver a gaming experience to a user. For example, a game console and an HMD may collaborate with a video server system 1220 to deliver a game viewed through the HMD. In one implementation, the game console receives a video stream from the video server system 1220, and the game console forwards the video stream or updates to the video stream to the HMD for rendering.

[0148] Client 1210 is configured to receive the video stream via network 1215. Network 1215 can be any type of communication network, including a telephone network, the Internet, a wireless network, a power line network, a local area network, a wide area network, a dedicated network, etc. In a typical implementation, the video stream is transmitted via a standard protocol such as TCP / IP or UDP / IP. Alternatively, the video stream is transmitted via a proprietary standard.

[0149] A typical example of a client 1210 is a personal computer, which includes a processor, non-volatile memory, a display, decoding logic, network communication capabilities, and input devices. The decoding logic may include hardware, firmware, and / or software stored on a computer-readable medium. Systems for decoding (and encoding) video streams are well known in the art and vary depending on the specific encoding scheme used.

[0150] Client 1210 may also include (but is not required to) a system configured to modify the received video. For example, the client may be configured to perform further rendering to overlay one video image on another video image, to crop the video image, etc. For example, client 1210 may be configured to receive various types of video frames, such as I frames, P frames, and B frames, and process these frames into images for display to the user. In some implementations, members of client 1210 are configured to perform further rendering, coloring, conversion to 3D, or similar operations on the video stream. Members of client 1210 are optionally configured to receive more than one audio or video stream. The input device of client 1210 may include, for example, a one-handed game controller, a two-handed game controller, a gesture recognition system, a gaze recognition system, a voice recognition system, a keyboard, a joystick, a pointing device, a force feedback device, a motion and / or position sensing device, a mouse, a touch screen, a neural interface, a camera, an input device not yet developed, and / or the like.

[0151] The video stream (and optionally, the audio stream) received by the client 1210 is generated and provided by the video server system 1220. As further described elsewhere herein, this video stream includes video frames (and the audio stream includes audio frames). Video frames are configured (e.g., they include pixel information in an appropriate data structure) to make a meaningful contribution to the image displayed to the user. As used herein, the term "video frame" is used to refer to a frame that primarily includes information configured to contribute to (e.g., implement) the image displayed to the user. Most of the teachings herein about "video frames" can also be applied to "audio frames."

[0152] Client 1210 is generally configured to receive input from a user. These inputs may include game commands that are configured to change the state of a video game or otherwise affect the process of playing the game. Game commands may be received using an input device and / or may be automatically generated by a computing instruction executed on client 1210. The received game commands are transmitted from client 1210 to video server system 1220 and / or game server 1225 via network 1215. For example, in some implementations, game commands are transmitted to game server 1225 via video server system 1220. In some implementations, a separate copy of the game command is transmitted from client 1210 to game server 1225 and video server system 1220. The communication of the game command optionally depends on the identity of the command. The game command is optionally transmitted from client 1210A through different routes or communication channels for providing audio or video streams to client 1210A.

[0153] The game server 1225 is optionally operated by an entity different from the video server system 1220. For example, the game server 1225 may be operated by the publisher of a multi-player game. In this example, the video server system 1220 is optionally viewed as a client by the game server 1225, and is optionally configured to appear as a prior art client executing a prior art game engine from the perspective of the game server 1225. The communication between the video server system 1220 and the game server 1225 is optionally via the network 1215. Therefore, the game server 1225 may be a prior art multi-player game server that sends game state information to multiple clients, one of which is the game server system 1220. The video server system 1220 may be configured to communicate with multiple instances of the game server 1225 simultaneously. For example, the video server system 1220 may be configured to provide multiple different video games to different users. Each of these different video games may be supported by a different game server 1225 and / or issued by a different entity. In some implementations, several geographically dispersed instances of the video server system 1220 are configured to provide game videos to multiple different users. Each of these instances of the video server system 1220 can communicate with the same instance of the game server 1225. Communication between the video server system 1220 and one or more game servers 1225 optionally occurs via a dedicated communication channel. For example, the video server system 1220 can be connected to the game server 1225 via a high-bandwidth channel dedicated to communication between the two systems.

[0154] The video server system 1220 includes at least a video source 1230, an I / O device 1245, a processor 1250, and a non-transitory storage device 1255. The video server system 1220 may include one computing device or be distributed among multiple computing devices. These computing devices may be connected via a communication system such as a local area network.

[0155] Video source 1230 is configured to provide video stream, for example, streaming video or a series of video frames forming moving pictures. In some implementations, video source 1230 includes a video game engine and rendering logic. The video game engine is configured to receive game commands from a player and maintain a copy of the state of the video game based on the received commands. This game state includes the position of the object in the game environment and the usual viewing angle. The game state may also include the properties, images, colors and / or textures of the object. The game state is usually maintained based on game rules and game commands such as moving, turning, attacking, setting focus, interacting, using, etc. The part of the game engine is optionally arranged in game server 1225. Game server 1225 can maintain a copy of the game state based on the game commands received from multiple players using geographically dispersed clients. In these cases, the game state is provided to video source 1230 by game server 1225, wherein a copy of the game state is stored and rendering is performed. Game server 1225 may receive game commands directly from client 1210 via network 1215 , and / or may receive game commands via video server system 1220 .

[0156] The video source 1230 typically includes rendering logic, such as hardware, firmware, and / or software stored on a computer-readable medium such as a storage device 1255. This rendering logic is configured to create video frames of a video stream based on the game state. All or a portion of the rendering logic may optionally be provided within a graphics processing unit (GPU). The rendering logic typically includes processing stages configured to determine three-dimensional spatial relationships between objects and / or apply appropriate textures, etc., based on the game state and viewing angle. The rendering logic generates raw video, which is then typically encoded prior to communication with the client 1210. For example, the raw video may be encoded according to Adobe Acrobat. The encoding process generates a video stream that is optionally packaged to be delivered to a decoder on a remote device. The video stream is characterized by a frame size and a frame rate. Typical frame sizes include 800×600, 1280×720 (e.g., 720p), 1024×768, but any other frame size can be used. The frame rate is the number of video frames per second. A video stream may include different types of video frames. For example, the H.264 standard includes "P" frames and "I" frames. An I frame includes information for refreshing all macroblocks / pixels on a display device, while a P frame includes information for refreshing a subset thereof. The data size of a P frame is typically smaller than an I frame. As used herein, the term "frame size" refers to the number of pixels within a frame. The term "frame data size" is used to refer to the number of bytes required to store a frame.

[0157] In an alternative implementation, video source 1230 includes a video recording device such as a camera. This camera can be used to generate delayed or live video that can be included in a video stream of a computer game. The resulting video stream optionally includes rendered images and images recorded using a static camera or a video camera. Video source 1230 may also include a storage device configured to store previously recorded videos to be included in the video stream. Video source 1230 may also include a motion or positioning sensing device configured to detect the motion or position of an object (e.g., a person), and logic configured to determine the game state or generate video based on the detected motion and / or position.

[0158] Video source 1230 is optionally configured to provide an overlay configured to be placed on other videos. For example, these overlays may include a command interface, login instructions, messages to the game player, images of other game players, video feeds of other game players (e.g., webcam videos). In implementations of client 1210A that include a touch screen interface or a gaze detection interface, the overlay may include a virtual keyboard, a joystick, a touchpad, and / or the like. In one example of an overlay, the player's voice is overlaid on an audio stream. Video source 1230 optionally also includes one or more audio sources.

[0159] In an implementation where the video server system 1220 is configured to maintain a game state based on input from more than one player, each player may have a different perspective, including the location and direction of viewing. The video source 1230 may optionally be configured to provide each player with a separate video stream based on their perspective. In addition, the video source 1230 may be configured to provide each client 1210 with a different frame size, frame data size, and / or encoding. The video source 1230 may optionally be configured to provide 3D video.

[0160] The I / O devices 1245 are configured to enable the video server system 1220 to send and / or receive information such as video, commands, information requests, game status, gaze information, device motion, device location, user motion, client identity, player identity, game commands, security information, audio, and / or the like. The I / O devices 1245 typically include communication hardware such as a network card or a modem. The I / O devices 1245 are configured to communicate with the game server 1225, the network 1215, and / or the client 1210.

[0161] Processor 1250 is configured to execute logic, e.g., software, included in the various components of video server system 1220 discussed herein. For example, processor 1250 may be programmed with software instructions to perform the functions of video source 1230, game server 1225, and / or client qualifier 1260. Video server system 1220 may optionally include more than one instance of processor 1250. Processor 1250 may also be programmed with software instructions to execute commands received by video server system 1220, or to coordinate the operation of various elements of game system 1200 discussed herein. Processor 1250 may include one or more hardware devices. Processor 1250 is an electronic processor.

[0162] Storage device 1255 includes non-temporary analog and / or digital storage devices. For example, storage device 1255 may include an analog storage device configured to store video frames. Storage device 1255 may include a computer-readable digital storage device, such as a hard drive, an optical drive, or a solid-state storage device. Storage device 1215 is configured (e.g., by an appropriate data structure or file system) to store video frames, artificial frames, video streams including both video frames and artificial frames, audio frames, audio streams, and / or the like. Storage device 1255 is optionally distributed in multiple devices. In some implementations, storage device 1255 is configured to store software components of video source 1230 discussed elsewhere herein. These components can be stored in a format that is ready to supply at any time.

[0163] The video server system 1220 optionally also includes a client qualifier 1260. The client qualifier 1260 is configured to remotely determine the capabilities of a client, such as client 1210A or 1210B. These capabilities may include the capabilities of the client 1210A itself and the capabilities of one or more communication channels between the client 1210A and the video server system 1220. For example, the client qualifier 1260 may be configured to test a communication channel through the network 1215.

[0164] The client qualifier 1260 can manually or automatically determine (e.g., discover) the capabilities of the client 1210A. Manual determination includes communicating with the user of the client 1210A and asking the user to provide capabilities. For example, in some implementations, the client qualifier 1260 is configured to display images, text, and / or the like within a browser of the client 1210A. In one implementation, the client 1210A is an HMD that includes a browser. In another implementation, the client 1210A is a game console with a browser that can be displayed on the HMD. The displayed object requests the user to input information such as the operating system, processor, video decoder type, network connection type, display resolution, etc. of the client 1210A. The information entered by the user is transmitted back to the client qualifier 1260.

[0165] The automatic determination may occur, for example, by executing an agent on the client 1210A and / or by sending a test video to the client 1210A. The agent may include computing instructions such as java scripts embedded in a web page or installed as an add-on. The agent is optionally provided by the client qualifier 1260. In various implementations, the agent may find the processing power of the client 1210A, the decoding and display capabilities of the client 1210A, the latency reliability and bandwidth of the communication channel between the client 1210A and the video server system 1220, the display type of the client 1210A, the firewall present on the client 1210A, the hardware of the client 1210A, the software executed on the client 1210A, the registry entries within the client 1210A, and / or the like.

[0166] The client qualifier 1260 includes hardware, firmware, and / or software stored on a computer-readable medium. The client qualifier 1260 is optionally disposed on a computing device separate from one or more other elements of the video server system 1220. For example, in some implementations, the client qualifier 1260 is configured to determine the characteristics of a communication channel between a client 1210 and more than one instance of the video server system 1220. In these implementations, the information discovered by the client qualifier can be used to determine which instance of the video server system 1220 is most suitable for delivering streaming video to one of the clients 1210.

[0167] Implementations of the present disclosure may be practiced with various computer system configurations, including handheld devices, microprocessor systems, microprocessor-based or programmable consumer electronics, minicomputers, mainframe computers, etc. The present disclosure may also be practiced in distributed computing environments where tasks are performed by remote processing devices that are linked through wired or wireless networks.

[0168] In view of the above implementations, it should be understood that the present disclosure may employ various computer-implemented operations involving data stored in a computer system. These operations are operations that require physical manipulation of physical quantities. Any operation that forms a part of the present disclosure described herein is a useful machine operation. The present disclosure also relates to a device or apparatus for performing these operations. The device may be specially constructed for the desired purpose, or the device may be a general-purpose computer selectively activated or configured by a computer program stored in a computer. Specifically, various general-purpose machines may be used together with a computer program written according to the teachings of this article, or more specialized devices may be more conveniently constructed to perform the desired operations.

[0169] The present disclosure may also be embodied as computer readable code on a computer readable medium. The computer readable medium is any data storage device that can store data that can then be read by a computer system. Examples of computer readable media include hard drives, network attached storage devices (NAS), read-only memories, random access memories, CD-ROMs, CD-Rs, CD-RWs, magnetic tapes, and other optical and non-optical data storage devices. Computer readable media may include computer readable tangible media distributed on network coupled computer systems so that computer readable code is stored and executed in a distributed manner.

[0170] Although the method operations are described in a particular order, it should be understood that other housekeeping operations may be performed between the operations, or the operations may be coordinated so that they occur at slightly different times, or the operations may be distributed in a system that allows processing operations to occur at various intervals associated with the processing as long as the processing of the covering operations is performed in the desired manner.

[0171] Although the foregoing disclosure has been described in considerable detail for the purpose of clear understanding, it is apparent that certain changes and modifications may be practiced within the scope of the appended claims. Therefore, the implementation of the present disclosure is to be considered illustrative rather than restrictive, and the disclosure is not limited to the details given herein, but may be modified within the scope and equivalent scope of the present disclosure.

Claims

1. A method comprising: receiving, from a client device via a network, a request for a virtual reality viewer to watch a live event via a head mounted display; Assigning the virtual reality spectator to a seat in a venue where the live event is occurring, wherein the seat is adjacent to at least one of a seat of a real spectator and a seat of another virtual reality spectator in the venue; receiving a plurality of video feeds from a plurality of cameras positioned in the venue, wherein, if the seat is adjacent to a seat of the real spectator, the video feeds include a video of the real spectator; and, if the seat is adjacent to a seat of another virtual reality spectator, the video feeds include a video of the another virtual reality spectator; accessing video processing parameters stored in association with the seat; selecting a video feed from among the video feeds using the video processing parameters and stitching the selected video feeds to generate a composite video providing a view of the venue from an angle substantially defined by the 3D position of the seats in the venue; The composite video is transmitted over the network to the client device for rendering to the head mounted display.

2. The method of claim 1 , wherein the video processing parameters identify which of the video feeds are selected for stitching, the video processing parameters being defined based on the 3D position of the seat associated with the video processing parameters and the 3D position of the camera providing the video feed.

3. The method of claim 1 , wherein assigning the virtual reality viewer to the seat comprises: Identifying an occupancy status of seats in the venue, wherein the occupancy status of a given seat indicates whether the given seat is occupied by a real spectator in the venue, wherein the seat assigned to the virtual reality spectator is a seat that is not occupied by a real spectator.

4. The method of claim 3, wherein the occupancy status of the given seat further indicates whether the given seat is occupied by another virtual reality viewer, wherein the seat allocated to the virtual reality viewer is a seat that is not occupied by another virtual reality viewer.

5. The method of claim 1 , wherein assigning the virtual reality viewer to the seat comprises: accessing a social graph of the virtual reality viewer; and selecting the seat based on proximity to a seat assigned to another virtual reality viewer who is a member of the social graph.

6. The method according to claim 1, further comprising: accessing audio processing parameters stored in association with the seat; generating audio data using the audio processing parameters that simulates listening from an angle substantially defined by the 3D position of the seat in the venue; The audio data is transmitted to the client device over the network.

7. The method of claim 6, wherein the audio processing parameters identify audio captured by one or more microphones in the venue to generate the audio data from the audio, and the audio processing parameters are defined based on the 3D positions of the seats and the 3D positions of the microphones associated with the audio processing parameters.

8. A non-transitory computer-readable medium having program instructions embodied thereon, the program instructions, when executed by at least one computer, causing the at least one computer to perform a method comprising: receiving, from a client device via a network, a request for a virtual reality viewer to watch a live event via a head mounted display; Assigning the virtual reality audience to seats in a venue where the live event is taking place, wherein: The seat is adjacent to at least one of a seat of a real spectator and a seat of another virtual reality spectator in the venue; receiving a plurality of video feeds from a plurality of cameras positioned in the venue, wherein, if the seat is adjacent to a seat of the real spectator, the video feeds include a video of the real spectator; and, if the seat is adjacent to a seat of another virtual reality spectator, the video feeds include a video of the another virtual reality spectator; accessing video processing parameters stored in association with the seat; selecting a video feed from among the video feeds using the video processing parameters and stitching the selected video feeds to generate a composite video providing a view of the venue from an angle substantially defined by the 3D position of the seats in the venue; The composite video is transmitted over the network to the client device for rendering to the head mounted display.

9. The non-transitory computer-readable medium of claim 8, wherein the video processing parameters identify which of the video feeds are selected for stitching, the video processing parameters being defined based on the 3D position of the seat associated with the video processing parameters and the 3D position of the camera providing the video feed.

10. The non-transitory computer-readable medium of claim 8, wherein assigning the virtual reality viewer to the seat comprises: Identifying an occupancy status of seats in the venue, wherein the occupancy status of a given seat indicates whether the given seat is occupied by a real spectator in the venue, wherein the seat assigned to the virtual reality spectator is a seat that is not occupied by a real spectator.

11. The non-transitory computer-readable medium of claim 10, wherein the occupancy status of the given seat further indicates whether the given seat is occupied by another virtual reality viewer, wherein the seat assigned to the virtual reality viewer is a seat that is not occupied by another virtual reality viewer.

12. The non-transitory computer-readable medium of claim 8, wherein assigning the virtual reality viewer to the seat comprises: accessing a social graph of the virtual reality viewer; and selecting the seat based on proximity to a seat assigned to another virtual reality viewer who is a member of the social graph.

13. The non-transitory computer-readable medium of claim 8, further comprising: accessing audio processing parameters stored in association with the seat; generating audio data using the audio processing parameters that simulates listening from an angle substantially defined by the 3D position of the seat in the venue; The audio data is transmitted to the client device over the network.

14. The non-transitory computer-readable medium of claim 13, wherein the audio processing parameters identify audio captured by one or more microphones in the venue to generate the audio data from the audio, the audio processing parameters being defined based on the 3D positions of the seats and the 3D positions of the microphones associated with the audio processing parameters.

15. A system comprising at least one computing device having at least one processor and at least one memory, wherein the at least one computing device is configured to perform the following operations: receiving, from a client device via a network, a request for a virtual reality viewer to watch a live event via a head mounted display; Assigning the virtual reality audience to seats in a venue where the live event is taking place, wherein: The seat is adjacent to at least one of a seat of a real spectator and a seat of another virtual reality spectator in the venue; receiving a plurality of video feeds from a plurality of cameras positioned in the venue, wherein, if the seat is adjacent to a seat of the real spectator, the video feeds include a video of the real spectator; and, if the seat is adjacent to a seat of another virtual reality spectator, the video feeds include a video of the another virtual reality spectator; accessing video processing parameters stored in association with the seat; selecting a video feed from among the video feeds using the video processing parameters and stitching the selected video feeds to generate a composite video providing a view of the venue from an angle substantially defined by the 3D position of the seats in the venue; The composite video is transmitted over the network to the client device for rendering to the head mounted display.

16. The system of claim 15, wherein the video processing parameters identify which of the video feeds are selected for stitching, the video processing parameters being defined based on the 3D position of the seat associated with the video processing parameters and the 3D position of the camera providing the video feed.

17. The system of claim 15, wherein assigning the virtual reality viewer to the seat comprises: Identifying an occupancy status of seats in the venue, wherein the occupancy status of a given seat indicates whether the given seat is occupied by a real spectator in the venue, wherein the seat allocated to the virtual reality spectator is a seat that is not occupied by a real spectator.

18. The system of claim 17, wherein the occupancy status of the given seat further indicates whether the given seat is occupied by another virtual reality viewer, wherein the seat assigned to the virtual reality viewer is a seat that is not occupied by another virtual reality viewer.

19. The system of claim 15, wherein assigning the virtual reality viewer to the seat comprises: accessing a social graph of the virtual reality viewer; and selecting the seat based on proximity to a seat assigned to another virtual reality viewer who is a member of the social graph.

20. The system of claim 15, further comprising: accessing audio processing parameters stored in association with the seat; generating audio data using the audio processing parameters that simulates listening from an angle substantially defined by the 3D position of the seat in the venue; The audio data is transmitted to the client device over the network.

21. The system of claim 20, wherein the audio processing parameters identify audio captured by one or more microphones in the venue to generate the audio data from the audio, the audio processing parameters being defined based on the 3D positions of the seats and the 3D positions of the microphones associated with the audio processing parameters.

Images