Collaborative immersive live action 360-degree video and virtual reality

Abstract

This disclosure relates to collaborative immersive live-action 360-degree video and virtual reality. A design for a collaborative live-action virtual reality experience is described. A client utilizes a 360-degree display unit to play a 360-degree video file, a corresponding audio file, and request user selections for annotations indicating the progress of the virtual reality experience. A script tree processes user input to determine subsequent 360-degree video, audio, and annotations to send to the client. In some embodiments, the progress of the virtual reality experience is indicated by annotation selections made by a second user, or by two users collaboratively participating in the virtual reality experience.

Description

[0001] Description of the case

[0002] This application is a divisional application of Chinese invention patent application 201780044392.3, filed on August 10, 2017. Technical Field

[0003] The present application relates generally to interactive and collaborative features of 360-degree video, and more particularly to creating a shared 360-degree interactive virtual reality environment, such as that used for training or instruction, playing games, etc. Background Art

[0004] Conventional web-based systems for storing and playing digital video files generally only support traditional playback of videos, where the video progresses in a linear manner without input from the viewer and without a system for recognizing and responding to the viewer's movements. These systems also do not allow multiple users to share the digital video experience from different perspectives, but rather provide a single perspective for playback for multiple users.

[0005] Virtual reality systems immerse their users in a 360-degree environment, allowing users to experience displayed content as if they were actually there. While conventional virtual reality systems foster an immersive experience by occupying the user's entire field of view and can allow users to engage with it in the form of video games, they are not traditionally compatible with two-dimensional video that does not respond to the user's viewpoint selection, and instead rely on dynamically generated audio / visual content that can be adjusted based on the user's viewpoint selection or the user's movement.

[0006] Furthermore, conventional VR systems allow users to experience VR works in an individual manner (e.g., with or without a headset), with the user's choice of viewpoint driving the experience. In other words, the user experience of VR works is typically not shareable among multiple users. Summary of the Invention

[0007] A method, system, and computer program product are disclosed for creating a collaborative live-action virtual reality experience that allows multiple users to simultaneously participate in creating a single interactive story from different perspectives. The story can be viewed on multiple clients, each associated with a 360-degree display unit. A user of a first client sends a request to a second client to initiate the virtual reality experience ("2-player mode"). In response to the user of the first client sending the request, the system accesses a first 360-degree media segment comprising a 360-degree video file, an audio file, and an annotation. The system displays the first 360-degree media segment on the first client and receives input from the first user in response to the annotation. In response to the first user providing the input, the system accesses a second 360-degree media segment corresponding to the annotation selection. The system then presents the second 360-degree media segment on the first client. In response to the first user providing input in response to the second annotation, the system accesses and displays a third 360-degree media segment. In other embodiments, the system accesses the 360-degree media segment in response to input from the second user or from both the first and second users.

[0008] In another implementation, a single user uses a client to send a request to the server to initiate a virtual reality experience ("1-player mode"). When the user of the client downloads the application, all 360-degree media segments stored in the video database and annotation database on the server are included in the download and stored on the client, so that the client does not need to connect to the network to start playing the 360-degree media segments. In the event that the user selects 1-player mode, all subsequent annotation selections are made by the user of the client. Alternatively, some annotations can be made automatically by the computer. Both 1-player and 2-player modes are contemplated, and the following description is only for the 2-player iteration. It will be clear to those skilled in the art that unless otherwise mentioned, the same functionality applies to the 1-player mode.

[0009] Aspects of the present disclosure provide a method for creating a shared interactive environment, comprising: sending a request to start a shared interactive experience computer program from a first client device, the first client device being associated with a first 360-degree display unit; receiving a response to initiating the shared interactive experience computer program; accessing a first 360-degree video file from a video database, the first 360-degree video file being linked to a plurality of 360-degree video files via a script tree, the script tree organizing the shared interactive experience based on user input responsive to annotations associated with the 360-degree video files; accessing a first annotation associated with the first 360-degree video file from the script tree; and using the first 360-degree display unit of the first client device to initiate the shared interactive experience computer program. Meta-visually presenting the first 360-degree video file and a first annotation; receiving user input in response to the first annotation; identifying a subsequent node in the script tree based on the user input; accessing a second 360-degree video file and a second annotation for the first client device based on the subsequent node, the second 360-degree video file and the second annotation being accessed based on the user input in response to the first annotation as determined by the script tree; and visually presenting the second 360-degree video file and the second annotation using a first 360-degree display unit of the first client device while simultaneously presenting a third 360-degree video file and a third annotation also associated with the subsequent node in the script tree on a second 360-degree display unit of the second client device.Aspects of the present disclosure provide a computer system comprising: a computer-readable storage medium comprising executable program code for creating a shared interactive environment, the computer program code: sending a request to start a shared interactive experience computer program from a first client device, the first client device being associated with a 360-degree display unit; receiving a response to initiating the shared interactive experience computer program; accessing a first 360-degree video file from a video database, the first 360-degree video file being linked to a plurality of 360-degree video files via a script tree, the script tree organizing the shared interactive experience based on user input responsive to annotations associated with the 360-degree video files; and accessing a first annotation associated with the first 360-degree video file from the script tree; visually presenting the first 360-degree video file and the first annotation using a first 360-degree display unit of the first client device; receiving user input in response to the first annotation; identifying a subsequent node in the script tree based on the user input; accessing a second 360-degree video file and a second annotation for the first client device based on the subsequent node, the second 360-degree video file and the second annotation being accessed based on the user input in response to the first annotation as determined by the script tree; and visually presenting the second 360-degree video file and the second annotation using the first 360-degree display unit of the first client device while simultaneously presenting a third 360-degree video file and a third annotation also associated with the subsequent node in the script tree using the second 360-degree display unit of the second client device.Aspects of the present disclosure provide a computer-readable medium (which may be a non-transitory computer-readable medium, but the present invention is not limited thereto) storing computer-executable instructions, the computer-executable instructions causing a device including a processor to perform operations in response to execution, the operations comprising: sending a request to start a shared interactive experience computer program from a first client device, the first client device being associated with a 360-degree display unit; receiving a response to initiating the shared interactive experience computer program; accessing a first 360-degree video file from a video database, the first 360-degree video file being linked to a plurality of 360-degree video files via a script tree, the script tree organizing the shared interactive experience based on user input in response to annotations associated with the first 360-degree video file; accessing a first 360-degree video file from the script tree, the first 360-degree video file being linked to a plurality of 360-degree video files via a script tree, the script tree organizing the shared interactive experience based on user input in response to annotations associated with the first 360-degree video file; and accessing a first 360-degree video file from the script tree. The invention relates to a method of visually presenting a first annotation associated with a 360-degree video file using a first 360-degree display unit of the first client device; receiving user input in response to the first annotation; identifying a subsequent node in the script tree based on the user input; accessing a second 360-degree video file and a second annotation for the first client device based on the subsequent node, the second 360-degree video file and the second annotation being accessed based on the user input in response to the first annotation as determined by the script tree; and visually presenting the second 360-degree video file and the second annotation using the first 360-degree display unit of the first client device while simultaneously presenting a third 360-degree video file and a third annotation also associated with the subsequent node in the script tree on the second 360-degree display unit of the second client device.

[0010] Other aspects of the present disclosure provide a computer-readable medium (which may be a non-transitory computer-readable medium, but the present disclosure is not limited thereto) storing computer-executable instructions, wherein the computer-executable instructions, in response to execution, cause a device including a processor to perform a method according to any aspect or embodiment described herein. Other aspects of the present disclosure provide a computer system configured to perform a method according to any aspect or embodiment described herein.

[0011] Aspects of the present disclosure therefore allow a virtual reality system or augmented display system to use video as input, thereby avoiding the need to provide dynamically generated audio-visual content. The absence of the need to dynamically generate audio-visual content can be particularly advantageous in situations where the virtual reality system uses a mobile device, as this eliminates the need to generate or transmit dynamically generated audio-visual content to the mobile device, which has lower processing power and / or storage capacity than wired devices. In addition, aspects of the present disclosure additionally or alternatively allow multiple users to drive their own experiences in an environment-based 360 video, while maintaining the ability to influence the experiences of others in the same "instance". Furthermore, aspects of the present disclosure can use 360 ​​video rather than a running rendered environment, thereby placing more of the experience burden on external bandwidth and the ability to stream video, while alleviating the processing burden on older mobile devices.

[0012] Furthermore, in situations where a virtual reality system allows for two or more users, it may be desirable for the audio / visual content provided to one user's device to be synchronized or substantially synchronized with the audio / visual content provided to another user's device, for example, when the users are interacting with each other in a virtual reality environment (e.g., in training audio / visual presentations). This is difficult to achieve when both users are using their own individual, dynamically generated content. This aspect of the present disclosure may additionally or alternatively allow audio / visual content to be more easily synchronized across two or more user devices.

[0013] The features and advantages described in this specification are not exhaustive, and in particular, many additional features and advantages will be apparent to those skilled in the art upon consideration of the drawings, the description, and the claims. Furthermore, it should be noted that the language used in the specification has been selected, in principle, for readability and instructional purposes, and may not have been selected to define or limit the disclosed subject matter. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1 is a block diagram of a system architecture for creating a collaborative live-action virtual reality experience, according to an embodiment.

[0015] Figures 2A-2C An example script tree is shown for organizing a collaborative live action virtual reality experience based on user input responsive to annotations, in accordance with an embodiment.

[0016] Figure 3 is a block diagram illustrating a server infrastructure designed to allow users to connect with each other to participate in a virtual reality experience, according to an embodiment.

[0017] Figure 4 is a flow diagram illustrating an example process for creating a collaborative live-action virtual reality experience, in accordance with an embodiment.

[0018] Figures 5A-5S Shown Figure 4 Use scenarios of the exemplary methods described in . DETAILED DESCRIPTION

[0019] Reference will now be made in detail to several embodiments, examples of which are illustrated in the accompanying drawings. Note that, where possible, similar or identical reference numerals may be used in the drawings and may indicate similar or identical functions. The accompanying drawings depict embodiments of the disclosed systems (and methods) for illustrative purposes only. Those skilled in the art will readily appreciate from the following description that alternative embodiments of the structures and methods described herein may be employed without departing from the principles described herein. Any features described herein in the context of one aspect or embodiment may be incorporated in any other aspect or embodiment described herein.

[0020] Exemplary Systems

[0021] Figure 1 1 is a block diagram of a system for creating a collaborative live action virtual reality experience. A plurality of users interact with a server facility 100 using clients 110A-B. Although only two clients 110A and 110B are shown, it is to be understood that a very large number (e.g., millions) of clients may be supported and may be in communication with the server facility 100 at any time. In order for the two clients 110 to share the virtual reality experience, client 110A may use web sockets to establish a connection with client 110B over a network 130 and exchange data packets with client 110B. Alternatively, client 110A may connect directly to client 110B using a wired connection or a short-range wireless communication standard such as Bluetooth or infrared.

[0022] The server facility 100 communicates with the clients 110A and 110B via a network 130, which is typically the Internet but can be any network, including but not limited to a LAN, MAN, WAN, mobile, wired or wireless network, private network or virtual private network. The clients 110 and the server facility 100 are connected to the network 130 via a network interface.

[0023] The client 110 includes a video player 140 and is coupled to a 360-degree display unit 150. Figure 1 , but the client 110 typically also includes a processor, a display device (or an output to a display device), local storage such as a hard disk or flash memory device in which the client 110 stores data used by the user when performing tasks, and a network interface for coupling to the server facility 100 via the network 130.

[0024] The client 110 can include a variety of different computing devices. Examples of the client 110 are personal computers, digital assistants, personal digital assistants, cellular phones, mobile phones, smartphones, or laptop computers. As will be apparent to those skilled in the art, the present invention is not limited to the devices listed above. In one embodiment, the client 110 also includes a speech recognition component that detects and interprets user selections in response to auditory input from the user of the client 110.

[0025] The 360-degree display unit 150 displays 360-degree video in a 360-degree display format, which is sometimes also referred to as "spherical video" or "immersive video". 360-degree video is a video recording captured by simultaneously recording the field of view in every direction. This is often done using an omnidirectional camera or a collection of multiple positioning cameras, however, other ways to record 360-degree video are expected, especially in the future. During playback, the user of the client 110 is able to control the viewing direction so that they can choose to view in any direction similar to a panorama, and the 360-degree display unit 150 will accept the selection and show the appropriate direction of the recorded video. In one embodiment, the 360-degree video is also a three-dimensional (3D) video and allows 3D visualization of the content (i.e., the depth of field moving through the z-axis or the perception of the z-axis).

[0026] Combined with the visual closure provided by the 360-degree display unit 150, the 360-degree display unit 150 can also be the entire range that the user can see so that they are completely visually immersed. That is, in some embodiments, all the user will be able to see is the 360-degree video, and the rest of their visual range is blocked by the 360-degree display unit 150. The 360-degree display unit 150 can similarly include audio speakers to allow the user to listen to the audio associated with the 360-degree video. The audio hardware of the 360-degree display unit 150 can be configured to play stereo sound in stereo or surround (e.g., 5 or more speaker sources) format.

[0027] The experience implemented for the user of client 110 and 360-degree display unit 150 can also be referred to as a "virtual reality experience" because the visual and auditory media simulate the user's physical presence and environment to create an artificial sensory experience for the user and allow user interaction. Therefore, the term "virtual reality experience" is used throughout this description to generally describe this experience. However, in various embodiments, this experience can be implemented using only 360-degree video, or it can be a more complete virtual reality experience.

[0028] In one embodiment, 360-degree display unit 150 is part of the same physical unit as client 110. Alternatively, 360-degree display unit 150 is a separate unit that is communicatively coupled to client 110 in a wired or wireless manner. An example of a 360-degree display unit 150 is a GOOGLE CARDBOARD unit; however, other 360-degree display units are also suitable and can be adapted to operate according to the techniques disclosed herein.

[0029] The 360-degree display unit 150 detects user input and transmits the input to the server facility 100 for analysis. The user input may be received in the form of a user-induced physical movement of the 360-degree display unit 150. The user input may also be received from a joystick or other controller held in the user's hand.

[0030] The video player 140 is used to control the presentation of 360-degree video and associated audio, and to control the presentation of text or audio annotations as part of the 360-degree experience. The combination of 360-degree video, audio, and annotations is referred to herein as a 360-degree media segment. The video player 140 is further used to control the sending and receiving of commands related to controlling the virtual reality experience. The video player 140 can be implemented in hardware or a combination of hardware and software. The video player 140 can be a standalone software application or a plug-in for another software application. The video player 140 is configured to cause the 360-degree display unit 150 to play 360-degree video files and audio files retrieved from the server facility 100 or stored within the client 110. When the video player 140 reaches a point in the 360-degree video where an annotation associated with the 360-degree video has been applied, the video player 140 causes the annotation to be presented by causing the 360-degree display unit 150 to overlay the annotation on the display screen and / or overlay the audio annotation on the audio track of the 360-degree video or audio file.

[0031] The video player 140 also includes user interface controls and, in some instances, corresponding application programming interfaces, which allow selection of 360-degree videos, start, stop, rewind, fast forward, and other navigation through the 360-degree video via input at the 360-degree display unit. Other types of user controls (e.g., buttons, keyboard controls) may also be used to control the playback and video format selection functions of the video player 140. The video player 140 also processes input via a joystick or movement of the 360-degree display unit 150 to guide selections as the user navigates through the virtual reality experience.

[0032] The server facility 100 facilitates the ability of clients 110A and 110B to connect with each other to participate in a collaborative live action virtual reality experience by providing 360-degree media segments as indicated by annotation selections received based on user input and as indicated by the logic flow of the virtual reality experience as further described below. In some instances, where 360-degree media segments are already stored locally on client 110, those segments will not need to be provided directly to client 110.

[0033] Organization of VR Experiences

[0034] Each 360-degree media segment contains video, audio, and annotations requesting one or more of the users of Clients 110A and 110B to make a decision, referred to herein as an annotation selection, which will indicate the progression of the experience. When one of the users makes an annotation selection by providing input—for example, by moving the 360-degree display unit 150—a user selection module 122 located in the server facility 100 or Client 110 interprets the user selection and instructs one or more video players 140 of the Clients 110 involved in the experience to load or obtain the 360-degree media segment associated with the selection. For example, a 360-degree media segment might include a video and audio file in which a player associated with Client device 110 is sitting in the driver's seat of a car, along with an annotation requesting the user to select whether to turn the car right or left. If the user of Client 110 selects the "left" option (i.e., a left annotation selection), the 360-degree display unit 150 will detect the input and transmit it to the user selection module 122, which will instruct the video player 140 to load the 360-degree media segment associated with the selection.

[0035] As described above, annotations can be displayed in the form of an interactive graphical interface overlaid on the displayed 360-degree video. For example, they can take the form of a menu-style interface, where the 360-degree display unit 150 displays several options via the annotations and then pauses the video to allow the user to select one of the options. Menu items / annotations allow for branching storylines within a virtual reality experience, where selecting an annotation leads to a 360-degree media segment, and clicking on a different annotation leads to a different 360-degree media segment. For example, annotations can be used in a game, where a user selects annotations indicating various forks in the storyline of a 360-degree media segment. If a first annotation selection is made, the user selection module 122 accesses a first 360-degree media segment from the server facility 100. If a second annotation selection is made, the user selection module 122 accesses a second 360-degree media segment from the server facility 100. Such a game is supported by multiple 360-degree media segments, each representing a destination for various jumps. Each jump in the game is represented as an entry in the organized hierarchical structure.

[0036] Figures 2A-2C An example script tree is shown illustrating a collection of 360-degree media segments organized and linked in a hierarchical structure. The script tree organizes a virtual reality experience. The script tree contains a series of nodes connected by a series of edges. At any given point during the experience, the script tree instructs a given video player 140 to play a given 360-degree media segment corresponding to each node in the script tree. Based on an annotation selection received at one of the clients 110 associated with the 360-degree media segment and node, the user selection module 122 accesses the script tree to determine which outgoing edge and node (often a child node) to take next. This in turn indicates which 360-degree media segment to play next to each client, and so on. In some cases, a node of the script tree has two or more outgoing edges, where each outgoing edge is associated with a selection that one or more users can make. For example, Figure 2A The node "gvr_01_start" in the script tree has four outgoing edges corresponding to the four different choices that can be made. In some cases, each of the multiple nodes of the script tree has at least one node that precedes it in the script tree and at least one node that follows it in the script tree (but for the start node and the leaf nodes corresponding to the end of the virtual reality environment (such as Figure 2B This is not the case for the leaf node “gvr_29_second_fail_end”).

[0037] The script tree begins with a root node representing the start of a collaborative 360-degree virtual reality experience for the users of client 110. In some embodiments, clients 110A and 110B continue navigating away from the root node until they reach a leaf node. For example, Figures 2A-2C Illustration of the script tree for a distracted driving public service announcement (PSA). Figure 2B gvr_29_second_fail_end is a leaf node because if the user of client 110A associated with the driver fails more than once, the virtual reality experience ends. In this way, the virtual reality experience can have different logical progressions of 360-degree media segments based on user selections and can have corresponding different endings.

[0038] Return Reference Figure 1 1 and client 110, video player 140 includes functionality to use a script tree structure and 360-degree media segments to implement a virtual reality experience for a user. User selection module 122 displays options on clients 110A and 110B for selecting a player through whose perspective the 360-degree media segment is to be viewed. For example, media player 140 displays a menu on client 110A that allows the user of client 110A to decide whether to view the 360-degree media segment as "Jess" or "Mike." If the user of client 110A selects "Mike," the user of client 110B will view the 360-degree video as "Jess." In one embodiment, three or more users of clients 110 participate in an interactive live-action virtual reality experience. The first user is the user of client 110A; the additional users may be other users of client 110 or a computer.

[0039] User selection module 122 receives user input from video player 140 and, in response to the input, compares the input to the script tree to determine which 360-degree media segment to retrieve from experience database 126 and annotation database 128. User selection module 122 then sends the 360-degree media segment to video player 140 along with instructions as to where to send the 360-degree media segment to clients 110A and 110B.

[0040] The script tree may contain multiple 360-degree media segments for each node, and the user selection module 122 may send different 360-degree media segments to each client 110 based on the player associated with each respective client 110 (e.g., different 360-degree media segments for the first client 110A vs. the second client 110B). When the user selection module 122 reaches a node in the script tree in response to user input, the user selection module 122 will make a different selection of the appropriate 360-degree media segment for each player, so that each user will hear and see something different during the virtual reality experience. Continuing with the example described above, if the user of client 110A selects Mike, the user selection module 122 will select the 360-degree media segment associated with player Mike at each node, and similarly for the other client 110B and player Jess.

[0041] In one embodiment, when no annotation is displayed on the client 110, the 360-degree display unit 150 detects user input in response to the user of the client 110 turning the 360-degree display unit 150 toward a point in the 360-degree field. The 360-degree display unit 150 transmits the input to the user selection module 122, which obtains the 360-degree media segment from the server facility 100 and instructs the video player 140 to play the 360-degree media segment.

[0042] The script tree also provides a node / edge structure that allows the user to backtrack to earlier points in the collaborative live-action virtual reality experience in response to appropriate selections of annotation choices based on the edge connections between nodes that enable such backtracking.

[0043] For example, assume that a user of client 110 selects a virtual reality experience for a puzzle from experience database 126, where the goal of the virtual reality experience is to search a house for clues that lead to the answer to the puzzle. Playback of the puzzle experience begins at the root node and navigates down the script tree along edges and nodes in response to the user selecting an annotation option indicating the story's direction. In one example, client 110 displays an annotation that asks the user of client 110 to select whether to walk into the library or the kitchen. The user of client 110 turns 360-degree display unit 150 to indicate the selection of the "library" annotation. In response to this annotation selection, user selection module 122 follows the edge associated with the "library" node and instructs video player 140 to play a 360-degree media segment in which a player associated with the user opens a door and walks into the library. As the player walks into the library, video player 140 displays an annotation that asks the user to select whether to search for clues on the desk or among wastepaper. The user of client 110 turns 360-degree display unit 150 to indicate the selection of the "desk" annotation. In response to the annotation selection, the user selection module 122 follows the edge associated with the "Desk" node and instructs the video player 140 to play a 360-degree media segment in which the player walks to the desk and searches for clues. When the video player 140 determines that the player has completed searching the desk, the video player 140 follows the backtracking edge node and returns the user of the client 110 to an earlier 360-degree media segment in which the user was asked to choose between searching the desk and searching the wastebasket. Returning the user to this point in the story allows the user to select a previously unselected annotation to change the progression of the storyline. Alternatively, the user can make the previous annotation selection again to repeat some aspects of the experience. The video player 140 can similarly employ backtracking to allow the user to search different rooms or floors of the house and navigate down the script tree to the leaf nodes associated with the puzzles that the user solved or failed to solve.

[0044] Server facilities

[0045] Figure 3 FIG is a block diagram of a server infrastructure 100 for a system for creating collaborative 360-degree virtual reality experiences. Figure 3 As shown, the server facility 100 includes a front-end server 120, a user selection module 122, an experience database 126, an annotation database 128, and a user database 130. Other conventional features such as firewalls, load balancers, application servers, fault-tolerant servers, site management tools, etc. are not shown to provide a clearer picture of the features of the system. Examples of suitable server facilities 100 for implementation of the system include YouTube TM and Google Video TMwebsite; other video hosting sites are also known and can be adapted to operate according to the teachings disclosed herein. It will be understood that the term "website" refers to any system and method for providing content and is not intended to be limited to systems that support content provided via the Internet or HTTP protocol. The various servers are implemented in a conventional manner, whether as a single piece of software or hardware or as multiple pieces of software or hardware, and can be coupled to the network 130 via the front-end server 120. In general, functions described as being performed on the server side in one embodiment can also be performed on the client side in other embodiments, where appropriate.

[0046] Front-end server 120 allows the user of client 110 to connect with users of other clients to initiate a virtual reality experience. Front-end server 120 displays a list of other users on client 110A and receives a selection from the user of client 110A of another user and a corresponding client 110B. Front-end server 120 then allows a request to be made to initiate the virtual reality experience. In one embodiment, the request includes a code that the user must enter into client 110B to join the virtual reality experience initiated by the other user.

[0047] The experience database 126 is used to store 360-degree media segments and associated metadata associated with the 360-degree media segments. 360-degree video and audio files have metadata associated with each file, such as video ID, video title, tags, and duration. The 360-degree media segments in the experience database 126 can be obtained from a variety of sources. Typically, they are acquired and stored in groups in the experience database 126 along with corresponding script trees, which collectively represent a cohesive virtual reality experience.

[0048] The annotation database 128 stores annotations associated with 360-degree media segments and the possible annotation selections allowed by the script tree. The annotation database 128 maintains an association between each annotation and the appropriate portion of the annotated 360-degree video. In one embodiment, for example, the annotation database 128 stores an identifier for the annotation type (e.g., a text box) along with any information associated with the type (e.g., a text title), a timestamp of the 360-degree video to which the annotation was applied (e.g., from time 01:05 to time 01:26), and an identifier for the 360-degree video annotated by the annotation.

[0049] User database 130 maintains a record of users associated with virtual reality experiences. Each individual user is assigned a user ID. The user ID can be based on any identifying information, such as the user's IP address, username, etc. To facilitate the initiation of new virtual reality experiences between users, user database 130 can contain, for a given user, a list of the identities of other users who are considered to be connections of the user.

[0050] Example procedures and use cases

[0051] Figure 4 is a flow chart illustrating an example process for creating a collaborative live action virtual reality experience in accordance with an embodiment. Figures 5A-5S This is demonstrated in the use case shown in .

[0052] Assume that Sean has a mobile device (client 110A) and a 360-degree display unit 150A communicatively coupled to his mobile device. Sean opens the application and selects the distracted driving PSA from the list of available 360-degree video files in the experience database 126 via the front-end server 120. Client 110A sends a request 410 to the server facility 100 to start the virtual reality experience. At 412, client 110A receives a response from the server facility 100 to initiate the selected program.

[0053] Steering Figure 5A , Sean was asked how many people he wanted to participate in the experience, and he chose two. Sean then Figure 5B 1. Sean is asked whether he wants to start a new session or join someone else's session. Sean selects "Start New Session" and is presented with a list of his connections included in user database 130. Sean selects Alison from his list of connections, and front-end server 120 displays a code that Alison must enter in order to join the session created by Sean.

[0054] Client 110A connects to Alison's device (client 110B) using web sockets and sends an invitation to client 110B, which is also coupled to 360-degree display unit 150B. In response to client 110A sending the invitation to client 110B, Alison enters the code to join the session. Figure 5C In the example, Sean is asked if he has Google Cardboard and he selects "yes". Figure 5DThe user selects "Go to VR Mode" in the VR mode and places the client 110A into the 360-degree display unit 150A to begin playback of the distracted driving PSA.

[0055] return Figure 4 In response to sending a response to initiating the collaborative live-action virtual experience to client 110A, user selection module 122 accesses 414 a first 360-degree video file and an associated first audio file from experience database 126. User selection module 122 associates 416 a first annotation with the first 360-degree video file and the first audio file, and retrieves the first annotation from annotation database 128. User selection module 122 then sends a 360-degree media segment containing the 360-degree video file, the associated audio file, and the annotation to video player 140 for display on client 110A.

[0056] At 418, the video player 140 presents the 360-degree media segment on the 360-degree display unit 150A. In some embodiments, the annotation includes only one annotation option. For example, on the client 110A, the first 360-degree media segment includes an instruction to start playing the selected video. Figures 5E-5G , Sean provides user input in the form of physical movement of 360-degree display unit 150A, indicating a selection of a "Play Now" annotation. 360-degree display unit 150A detects this input 420 and transmits it to user selection module 122 for interpretation. User selection module 122 compares this input with the script tree to determine which 360-degree media segment to retrieve from the experience database in response to the "Play Now" input. User selection module 422 then retrieves a second 360-degree video file and associated audio file from experience database 126 and a second annotation 424 from annotation database 128. User selection module 122 then instructs video player 140 to begin playing the second 360-degree media segment on client 110A.

[0057] The video player 140 begins playing the second 360-degree media segment 426. In other embodiments, the annotation includes two or more annotation options. For example, Figure 5H In

[0045] , the video player 140 displays an annotation on the 360-degree display unit 150A asking Sean to choose whether he wants to experience the distracted driving PSA from Jess's or Mike's perspective. Figure 5I , Sean provides a user input indicating a selection of the "Jess" annotation. The user selection module 122 interprets the input and instructs the video player 140 to begin playing the 360-degree media segment associated with the selected node.

[0058] exist Figure 5JIn the video player 140, the third 360-degree media segment begins playing. Assuming Sean selects the "Mike" annotation, the video player 140 presents an annotation asking Sean whether to select "Follow Jasmine" or "Go Jess's Way." Figure 5K In the example, Sean provides user input and the user selection module 122 selects Figure 5L This is interpreted as selection of the “Walk Jess’s Way” node, instructing the video player 140 to start playing the corresponding 360-degree media segment.

[0059] Figure 5M The figure shows a 360-degree media segment with four annotation options. Sean is asked to choose whether to play digital dance music, pop, disco, or hip-hop music. Figure 5N In the Figure 5O . The user selection module 122 interprets the input and instructs the video player 140 to play the 360-degree media segment corresponding to the selected node. In one embodiment, selection of the annotation option causes the entire 360-degree media segment associated with the selected option to play immediately. In another embodiment, selection of the annotation option results in a delayed response, causing the video player 140 to delay playing a portion of the 360-degree media segment. For example, while the video player 140 may start the 360-degree video file associated with Sean's selection of popular music, the video player 140 may delay playing the corresponding audio file containing the popular music for a specified period of time.

[0060] In some embodiments, users of clients 110A and 110B are presented with annotations and asked to select from displayed annotation options. Figure 5P In , the video player 140 displays annotations with two annotation options on clients 110A and 110B. Both Sean and Alison are asked to choose between "Kiss" and "No kiss". Figure 5Q , Sean provides user input indicating a selection of the "Kiss" node. However, because the third annotation requires input from both players, the user selection module 122 will instruct the video player 140A to play the 360-degree media segment associated with the "Kiss" node only if Alison also selects the "Kiss" annotation option.

[0061] exist Figure 5RIn the example, video player 140 displays a selection of the "kiss" annotation option, indicating that Alison also selected the "kiss" annotation option. Client 110A receives the data packet indicating Alison's selection from client 110B and instructs video players 140A and 140B to play the 360-degree media segments corresponding to the two users' annotation selections. Because the script tree contains multiple videos for each node, user selection module 122 will send different videos to clients 110A and 110B representing each player's perspective of the same event in the virtual reality experience.

[0062] Figure 5S The distracted driving PSA ends. Video player 140 plays the final 360-degree media segment containing the annotation that says "You made it." This final annotation does not prompt Sean or Alison to select from multiple annotation options, but rather indicates the conclusion of the collaborative live-action virtual reality experience.

[0063] Additional considerations

[0064] The present invention has been described in particular detail with respect to various embodiments, and those skilled in the art will appreciate that the present invention may be practiced in other embodiments. In addition, those skilled in the art will appreciate the following aspects of the present disclosure. First, the specific naming of components, capitalization of terms, attributes, data structures, or any other procedural or structural aspects are not mandatory or important, and the mechanisms of the present invention or its features may have different names, formats, or protocols. Second, the system may be implemented as described via a combination of hardware and software, or entirely in hardware elements. Third, the specific functional divisions between the various system components described herein are exemplary only and not mandatory; the functions performed by a single system component may alternatively be performed by multiple components, and the functions performed by multiple components may alternatively be performed by a single component.

[0065] Some portions of the above description provide features of the present invention in terms of algorithms and symbolic representations of operations performed on information. These algorithmic descriptions and representations are the most effective means for those skilled in the art to convey the essence of their work to other persons skilled in the art. Although described functionally, computationally, or logically, these operations are understood to be implemented by computer programs or equivalent circuits, microcodes, etc. In addition, it has also been demonstrated many times that referring to the deployment of these operations as modules is convenient without loss of generality. The described operations and their associated modules can be embodied in software, firmware, or hardware.

[0066] Furthermore, the terms used to describe various quantities, data values, and operations are to be understood as being associated with the appropriate physical quantities and are merely convenient labels applied to these quantities. Unless otherwise specifically noted as being clear from the following discussion, it will be appreciated that throughout this description, discussions using terms such as "processing" or "operating" or "calculating" or "determining" refer to the action and processing by a computing system or similar electronic computing device of manipulating or transforming data represented as physical (electronic) quantities within a computing system memory or register or other such information storage, transmission, or display device.

[0067] Finally, it should be noted that the language used in the specification has been selected in principle for the purpose of readability and teaching, and has not been selected to define or limit the subject matter of the invention. Therefore, the disclosure of the present invention is intended to be illustrative rather than limiting the scope of the invention as set forth in the following claims.

Claims

1. A method for creating a shared interactive gaming experience, comprising: sending a request to a server over a network from a first client device associated with a first 360-degree display unit to begin a shared interactive gaming experience; receiving, at the first client device, a response to the request, the response allowing the first client device to initiate the shared interactive gaming experience; connecting with a second client device via the network to collaborate on the shared interactive gaming experience, the second client device being associated with a second 360-degree display unit; Retrieving a first 360-degree video file and a first annotation associated with the first 360-degree video file from a data store of the server based on a script tree in the data store, wherein the script tree represents a plurality of flows of the shared interactive gaming experience, each flow comprising a plurality of nodes, each of the plurality of nodes corresponding to one or more 360-degree video files and associated annotations; visually presenting the first 360-degree video file and the first annotation using the first 360-degree display unit of the first client device, the first annotation including one or more options for selection by a user of the first client device; receiving, at the first client device, user input selecting one of the one or more options associated with the first annotation, wherein the user input is caused by movement of the first 360-degree display unit when the user of the first 360-degree display unit selects the one of the one or more options associated with the first annotation; Obtaining a second 360-degree video file and associated second annotation corresponding to a subsequent node in the script tree of the first client device; and The second 360-degree video file and the second annotation are visually presented using the first 360-degree display unit of the first client device, while the second 360-degree display unit of the second client device is presenting a third 360-degree video file and an associated third annotation that also corresponds to the subsequent node in the script tree, wherein the second 360-degree video file and the third 360-degree video file provide the shared interactive experience from different perspectives.

2. The method according to claim 1, wherein Each of the plurality of nodes of the script tree has at least one node preceding the node and at least one node following the node.

3. The method of claim 1 , further comprising visually presenting a fourth 360-degree video file and a fourth annotation using the first 360-degree display unit of the first client device in response to user input made by the first client device in response to the second 360-degree video file and the second annotation.

4. The method according to claim 3, wherein: A node of the script tree has two or more outgoing edges, each outgoing edge being associated with one or more selections that a user can make.

5. The method according to claim 1, wherein The plurality of nodes of the script tree correspond to a plurality of 360-degree video files, and wherein a different 360-degree video file is presented to each client at each of the plurality of nodes of the script tree.

6. The method according to claim 1, further comprising: visually presenting a fourth 360-degree video file and a fourth annotation using the first 360-degree display unit of the first client device in response to user input made by the first client device in response to the second 360-degree video file and the second annotation; receiving user input in response to the fourth annotation; as well as The first 360-degree video file and the first annotation are visually presented using the first 360-degree display unit of the first client device in response to the received user input responsive to the fourth annotation.

7. The method according to claim 1, wherein The second 360-degree video file and the second annotations are accessed for the first client device based on the user of the first client device having selected to become a first player, and wherein the third 360-degree video file and the third annotations are accessed for the second client device based on the user of the second client device having selected to become a second player.

8. The method according to claim 1, wherein The second annotation and the third annotation are the same.

9. The method according to claim 1, wherein The second annotation and the third annotation are different.

10. A computer system comprising: one or more processors; as well as a memory storing one or more programs for execution by the one or more processors, the one or more programs comprising instructions for: sending a request to a server over a network from a first client device associated with a first 360-degree display unit to begin a shared interactive gaming experience; receiving, at the first client device, a response to the request, the response allowing the first client device to initiate the shared interactive gaming experience; connecting with a second client device via the network to collaborate on the shared interactive gaming experience, the second client device being associated with a second 360-degree display unit; Retrieving a first 360-degree video file and a first annotation associated with the first 360-degree video file from a data store of the server based on a script tree in the data store, wherein the script tree represents a plurality of flows of the shared interactive gaming experience, each flow comprising a plurality of nodes, each of the plurality of nodes corresponding to one or more 360-degree video files and associated annotations; visually presenting the first 360-degree video file and the first annotation using the first 360-degree display unit of the first client device, the first annotation including one or more options for selection by a user of the first client device; receiving, at the first client device, user input selecting one of the one or more options associated with the first annotation, wherein the user input is caused by movement of the first 360-degree display unit when the user of the first 360-degree display unit selects the one of the one or more options associated with the first annotation; Obtaining a second 360-degree video file and associated second annotation corresponding to a subsequent node in the script tree of the first client device; and A second 360-degree video file and the second annotation are visually presented using the first 360-degree display unit of the first client device while the second 360-degree display unit of the second client device is presenting a third 360-degree video file and associated third annotation that also corresponds to the subsequent node in the script tree, wherein the second 360-degree video file and the third 360-degree video file provide the shared interactive gaming experience from different perspectives.

11. The system according to claim 10, wherein: Each of the plurality of nodes of the script tree has at least one node preceding the node and at least one node following the node.

12. The system according to claim 10, wherein: The plurality of nodes of the script tree correspond to a plurality of 360-degree video files, and wherein a different 360-degree video file is presented to each client at each of the plurality of nodes of the script tree.

13. The system of claim 10, further comprising: visually presenting a fourth 360-degree video file and a fourth annotation using the first 360-degree display unit of the first client device in response to user input made by the first client device in response to the second 360-degree video file and the second annotation; receiving user input in response to the fourth annotation; as well as The first 360-degree video file and the first annotation are visually presented using the first 360-degree display unit of the first client device in response to the received user input responsive to the fourth annotation.

14. The system according to claim 10, wherein: The second 360-degree video file and the second annotations are accessed for the first client device based on the user of the first client device having selected to become a first player, and wherein the third 360-degree video file and the third annotations are accessed for the second client device based on the user of the second client device having selected to become a second player.

15. The system according to claim 10, wherein: The second annotation and the third annotation are the same.

16. A non-transitory computer-readable medium storing computer-executable instructions that, in response to being executed, cause a device comprising a processor to perform operations comprising: sending a request to a server over a network from a first client device associated with a first 360-degree display unit to begin a shared interactive gaming experience; receiving, at the first client device, a response to the request, the response allowing the first client device to initiate the shared interactive gaming experience; connecting with a second client device via the network to collaborate on the shared interactive gaming experience, the second client device being associated with a second 360-degree display unit; Retrieving a first 360-degree video file and a first annotation associated with the first 360-degree video file from a data store of the server based on a script tree in the data store, wherein the script tree represents a plurality of flows of the shared interactive gaming experience, each flow comprising a plurality of nodes, each of the plurality of nodes corresponding to one or more 360-degree video files and associated annotations; visually presenting the first 360-degree video file and the first annotation using the first 360-degree display unit of the first client device, the first annotation including one or more options for selection by a user of the first client device; receiving, at the first client device, user input selecting one of the one or more options associated with the first annotation, wherein the user input is caused by movement of the first 360-degree display unit when the user of the first 360-degree display unit selects the one of the one or more options associated with the first annotation; Obtaining a second 360-degree video file and associated second annotation corresponding to a subsequent node in the script tree of the first client device; and A second 360-degree video file and the second annotation are visually presented using the first 360-degree display unit of the first client device while the second 360-degree display unit of the second client device is presenting a third 360-degree video file and associated third annotation that also corresponds to the subsequent node in the script tree, wherein the second 360-degree video file and the third 360-degree video file provide the shared interactive gaming experience from different perspectives.

17. The computer-readable medium of claim 16, wherein: Each of the plurality of nodes of the script tree has at least one node preceding the node and at least one node following the node.

18. The computer-readable medium of claim 16, wherein: The plurality of nodes of the script tree correspond to a plurality of 360-degree video files, and wherein a different 360-degree video file is presented to each client at each of the plurality of nodes of the script tree.

19. The computer-readable medium of claim 16, further comprising instructions for: visually presenting a fourth 360-degree video file and a fourth annotation using the first 360-degree display unit of the first client device in response to user input made by the first client device in response to the second 360-degree video file and the second annotation; receiving user input in response to the fourth annotation; as well as The first 360-degree video file and the first annotation are visually presented using the first 360-degree display unit of the first client device in response to the received user input responsive to the fourth annotation.

20. The computer-readable medium of claim 16, wherein: The second 360-degree video file and the second annotations are accessed for the first client device based on the user of the first client device having selected to become a first player, and wherein the third 360-degree video file and the third annotations are accessed for the second client device based on the user of the second client device having selected to become a second player.

Images