System and method for creating and managing virtual studios

Abstract

In at least one embodiment, a system for controlling aspects of a virtual concert is provided. The apparatus includes one or more controllers and at least one computing device. The one or more controllers are located at a venue and are configured to control features of a live performance at the venue based on at least one first signal. The at least one computing device is programmed to receive a second signal directly from a user remote from the venue, the second signal indicating a command to control at least a portion of the live performance, and to transmit the at least one first signal to the one or more controllers for controlling the features of the live performance.

Description

Technical Field

[0001] Cross - reference to Related Applications This application claims the benefit of U.S. Provisional Application No. 63 / 053,318, filed on July 17, 2020, the entire disclosure of which is incorporated herein by reference.

Background Art

[0002] Aspects disclosed herein generally relate to systems and methods for creating and managing virtual - counterpart studios. In certain embodiments, aspects disclosed herein may correspond to systems and methods for creating and managing virtual - counterpart studios that allow a user to remotely interact via connected devices that bring actual output into the studio. Such aspects and other aspects are described in more detail below.

[0003] A user may not be able to attend a live concert at an actual concert venue. For example, a user may be unable to attend a live concert for various reasons. For example, due to the recent pandemic, it has been prohibited for large numbers of people to gather in small spaces for concerts. Further, even without pandemic concerns, concert attendees (or fans) may be unable to attend a concert due to their location or the distance between the concert venue and the fan's location. It may be desirable to enable a user to control such aspects while experiencing various aspects of a live performance based on the fan's preferences from a location far away from the actual live - performance location.

Summary of the Invention

Means for Solving the Problems

[0004] In at least one embodiment, a system is provided for controlling aspects of a virtual concert. This system includes one or more controllers and at least one computing device. One or more controllers are located in a venue and are configured to control features of a live performance at the venue based on at least one first signal. At least one computing device is programmed to receive a second signal indicating a command to control at least a portion of the live performance directly from a user far from the venue, and to transmit at least one first signal to one or more controllers for controlling features of the live performance.

[0005] In at least another embodiment, a method is provided for controlling the nature of a virtual concert. This method includes, via one or more controllers located in the venue, controlling the characteristics of a live performance at the venue based on at least one first signal, and receiving, on at least one computing device, a second signal indicating commands to control at least a portion of the live performance directly from a user located far from the venue. The method further includes transmitting at least one first signal to one or more controllers to control the characteristics of the live performance.

[0006] In at least another embodiment, a computer program product embodied in a non-temporary computer-readable medium is programmed for a form of virtual concert controller. The computer program product includes instructions for controlling features of a live performance at a venue based on at least one first signal via one or more controllers located within the venue, and instructions for receiving, on at least one computing device, a second signal indicating commands to control at least a portion of the live performance directly from a user located far from the venue. The computer program product includes instructions for transmitting at least one first signal to one or more controllers to control features of the live performance. This specification also provides, for example, the following: (Item 1) A system for controlling the nature of a virtual concert, comprising one or more controllers placed within a venue and configured to control the characteristics of a live performance in the venue based on at least one first signal, Equipped with at least one computing device, The at least one computing device is A second signal indicating commands to control at least a portion of the live performance is received directly from a user far away from the venue. The system is programmed to transmit at least one first signal to one or more controllers for controlling the characteristics of the live performance. (Item 2) The system according to item 1, wherein the one or more controllers include a camera controller, the camera controller is configured to control one or more cameras positioned in the venue to move to a desired camera angle, and to transmit a first video stream of the live performance based on the desired camera angle to the at least one computing device. (Item 3) The system according to item 2, wherein one or more cameras are positioned directly on the performer at the venue, or directly on the performer's instrument. (Item 4) The system according to item 3, wherein the camera controller is further configured to activate one or more cameras positioned directly on the performer or on the instrument, and to provide a second video stream corresponding to the captured video stream of the performer or the performer's instrument. (Item 5) The system according to item 2, wherein the at least one computing device is further configured to display a mapping of the cameras arranged throughout the venue, to allow a user to select items on the mapping, and to control the cameras arranged in the venue. (Item 6) The system according to item 2, wherein the one or more controllers include a lighting controller configured to control aspects related to lighting arranged in the live performance based on the at least one first signal, and the camera controller is further configured to transmit a captured video stream showing desired changes to the lighting in the live performance to at least one computing device. (Item 7) The lighting system arranged in the live performance includes one or more of the following: spotlights, strobe lights, stage lights, and animations, as described in item 6. (Item 8) The system according to item 6, wherein the at least one computing device is further configured to display a mapping of the lighting arranged throughout the venue, to allow a user to select items on the mapping, and to control the lighting arranged in the venue. (Item 9) The system according to item 2, wherein the one or more controllers include a robot controller that controls at least one mechanical prop installed in the live performance based on the at least one first signal, and the camera controller is further configured to transmit a captured video stream to the at least one computing device corresponding to the movement of the at least one mechanical prop based on the at least one first signal. (Item 10) The system according to item 9, wherein the at least one computing device is further configured to display a mapping of the at least one mechanical prop installed in the venue, to allow a user to select an item on the mapping, and to control the at least one mechanical prop in the venue. (Item 11) The system according to item 1, wherein the one or more controllers include a first controller configured to control one or more first items corresponding to one or more of the following in the live performance: pyrotechnics, confetti cannons, video / audio projection onto a screen, audio microphone equipment, and amplification, based on the at least one first signal. (Item 12) The audio microphone system according to item 11, comprising one or more binaural microphones placed in the venue to capture the atmosphere of the audience in the venue. (Item 13) The system according to item 11, wherein the at least one computing device is further configured to display a mapping of the one or more first items placed in the venue based on the at least one first signal, allowing the user to select one or more of the following for the live performance: pyrotechnics, confetti cannons, video / audio projection onto a screen, audio microphone equipment, and amplification. (Item 14) The system according to item 1, wherein the at least one computing device is further configured to allow the user to select one or more of the following: cheer credits, ticket tiers, and playlist events. (Item 15) The cheering credits correspond to one or more of the following: requests to change stage lighting, voting for songs to be played at the venue, sending personalized messages to performers during the live performance, and requests for personalized messages to performers, as described in item 14. (Item 16) The aforementioned ticket hierarchy is, General admission tier for remotely viewing live performances on at least one computing device, A reserved seating tier in which the user can obtain HiFi audio quality while being located far from the venue. A front row seating tier that blocks the display of advertisements while viewing the live performance on the at least one computing device, and Backstage pass tiers that offer signed items from performers on stage, A system described in item 14 that corresponds to any one or more of the following. (Item 17) The aforementioned playlist event is, A super fun shootout event that allows users of multiple computing devices to compete against each other. A light event that allows a user of at least one computing device to select an option on each of the computing devices to trigger a light representing the user holding the lighter during the live performance. A decibel meter event that allows the user of at least one computing device to tap an interface placed thereon within a predetermined time to simulate a fan repeatedly tapping during a live performance to increase the decibel level brought about by the audience, and A live voting event that allows viewers to vote for performers to perform specific actions. A system described in item 14 that corresponds to any one or more of the following. (Item 18) The system according to item 1, further comprising one or more controllers and a server operably coupled to at least one computing device, wherein the server is configured to determine user reputation among multiple users of the multiple computing devices. (Item 19) A method for controlling the manner of a virtual concert, Controlling the characteristics of the live performance at the venue based on at least one first signal via one or more controllers placed within the venue, In at least one computing device, a second signal indicating commands for controlling at least a portion of the live performance is received directly from a user located far from the venue. To transmit the at least one first signal to one or more controllers to control the characteristics of the live performance, The method, including the method described above. (Item 20) A computer program product embodied in a non-temporary computer-readable medium programmed for a virtual concert controller, Commands for controlling the characteristics of the live performance at the venue based on at least one first signal, via one or more controllers placed within the venue, Instructions for receiving a second signal indicating commands to control at least a portion of the live performance directly from a user located far from the venue, in at least one computing device, Sending the at least one first signal to the one or more controllers as a command to control the features of the live performance, The computer program product including the above.

[0007] Embodiments of this disclosure are described in detail in the appended claims. However, other features of various embodiments will become clearer and best understood by referring to the following detailed description in conjunction with the appended drawings. [Brief explanation of the drawing]

[0008] [Figure 1] This document presents a system for creating and managing a virtual studio or live performance, according to one embodiment. [Figure 2] One embodiment of a method for controlling one or more cameras for a virtual-enabled studio or live performance is shown. [Figure 3] This document describes a method for controlling lighting in a virtual-enabled studio or for live performances, according to one embodiment. [Figure 4] One embodiment of a method for controlling one or more props for a virtual-enabled studio or live performance is presented. [Figure 5] One embodiment of a method for controlling activities for a virtual-enabled studio or live performance is presented. [Figure 6] This embodiment describes a method for determining prestige between a first user and a second user when conflicting commands are provided for controlling aspects of a virtual-enabled stereo or live performance. [Figure 7] An example of cheer credits that can be issued by the system shown in Figure 1, according to one embodiment, is shown. [Figure 8] An example of additional cheer credits that can be issued by the system in Figure 1, according to one embodiment, is shown. [Figure 9] An example of a ticket hierarchy that can be issued by the system in Figure 1, according to one embodiment, is shown. [Figure 10] An example of an exclusive feature that may be issued by the system in Figure 1, according to one embodiment, is shown. [Figure 11] An example of a special offer that may be issued by the system in Figure 1, according to one embodiment, is shown. [Figure 12] An example of a playlist event that may be issued by the system in Figure 1, according to one embodiment, is shown. [Figure 13] An example of a playlist event that may be issued by the system in Figure 1, according to one embodiment, is shown. [Figure 14] An exemplary user interface on one or more computing devices of the system in Figure 1, according to one embodiment, is shown. [Figure 15] This invention illustrates a system for remotely creating audio / video mixes and masters for live audio and video streams. [Figure 16] Figure 15 shows an interface screen provided by the computing device of the system according to an embodiment. [Figure 17] This document describes a method for temporally aligning audio and video streams from a live performance according to one embodiment. [Figure 18] This document describes a method for providing a "picture-in-picture stream" for live performances according to one embodiment. [Modes for carrying out the invention]

[0009] Where necessary, detailed embodiments of the present invention are disclosed herein, but it should be understood that the disclosed embodiments are merely illustrative examples of the present invention, which may be embodied in various and alternative forms. The figures are not necessarily to scale, and some features may be exaggerated or minimized to illustrate the details of certain components. Accordingly, the specific structural and functional details disclosed herein should not be construed as limiting, but rather as representative grounds to teach those skilled in the art how to utilize the present invention in various ways.

[0010] It is recognized that at least one controller (or at least one processor) disclosed herein may include various microprocessors, integrated circuits, memory devices (e.g., flash memory, random access memory (RAM), read-only memory (ROM), electrically programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or other suitable variations), and software that cooperates with each other to perform the operations(s) disclosed herein. In addition, any disclosed controller may utilize any one or more microprocessors to execute a computer program embodied in a non-temporary computer-readable medium programmed to perform the various functions disclosed herein. Furthermore, any controller(s) provided herein may include a housing and various numbers of microprocessors, integrated circuits, and memory devices (e.g., flash memory, random access memory (RAM), read-only memory (ROM), electrically programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM)) arranged inside the housing. The disclosed controller(s) also include hardware-based inputs and outputs for sending and receiving data between other hardware-based devices as discussed herein.

[0011] A system and method for creating and managing virtual studios or live performances. The embodiments disclosed herein generally provide an entire music venue as a live streaming studio, including lighting, audio microphone equipment, amplification, video recording, projection, etc., to deliver an online virtual concert experience. This technology and its embodiments are not limited to virtual concerts and can be used for any live streaming event. Remote users may control and trigger a number of different physical devices within the venue by issuing commands online via chat. These commands trigger real-world events occurring in real time at the location of the stream (or live performance).

[0012] At the heart of this technology is a main server ("Server") (or at least one controller) with a number of nodes provided. The Server comprises any number of microprocessors capable of executing instructions to perform any of the functions described herein. In one example, the Server may be any type of online connected computer device capable of sending commands (e.g., messages) to any number of nodes. Each node may include one or more microcontrollers, computers, and mobile devices (e.g., mobile phones, tablets) configured to interpret commands (messages) from the Server and execute commands associated with the commands (or messages). The Server may interpret keywords sent in a chat (e.g., Internet Relay Chart (IRC)) associated with a streaming service (e.g., a chatbot). This service may interpret messages sent by users who want to trigger events occurring in a concert venue or studio where an event is taking place. In another embodiment, a user triggers an event via a user interface of a computing device, and the Server interprets such responses or commands and plays them back in the streaming venue. The Server includes a database for collecting responses or commands sent to the venue. The Server may also aggregate the data and send it to the venue as commands, and send the data back to other computing devices. One example might involve sending an interaction (e.g., emojis, cheers, votes, or any other type of user interaction) from a first computing device to display on a venue screen. The server could also send the interaction to other computing devices so that the user can view it on those devices if they wish.

[0013] An online currency for the event can be created and exchanged for local currency. Once exchanged, users may use this currency during the event to trigger desired events associated with the live performance. Users may also be granted certain tiers of prestige depending on the level of access they purchase when joining the video stream. These prestige levels grant users access to additional events, which they are allowed to trigger at the studio / venue designated as the live venue.

[0014] When a user attempts to trigger a desired event during a live performance, the server may determine whether the user has the appropriate prestige and balance to trigger such a desired event. If so, the server may send an appropriate message to trigger the event, or send a message back to the user informing them that they lack the necessary prestige, balance, or both to trigger such an event. This currency may be awarded to the user for various events. The currency may also be used to purchase various items from a dedicated store available before, during, or after an event. For example, such purchases may only be permitted during the event.

[0015] The location where the live stream is held (e.g., the live stream venue) can be equipped with multiple nodes (e.g., multiple electronic nodes) that can be controlled by a remote user. These nodes may include, but are not limited to, pyrotechnics, cameras, robots, hydraulic systems, stage lighting, audience lighting, stage lighting animations, audio, animations rendered on the stream and trigger animations placed on large screens visible at the venue, text displayed on screens within the venue, and emojis on screens within the venue. Nodes may be equipped with computers or mobile devices (e.g., phones, tablets, etc.), each containing any number of microcontrollers. The microcontrollers translate messages / commands from the server and convert such commands into digital or analog messages (e.g., serial, digital multiplex (DMX), Inter-Integrated Circuit (I2C), User Datagram Protocol (UDP), JavaScript® Object Notation (JSON), relays, voltage, current, resistance, capacitance, inductance, magnetic fields, and electric fields, etc.) to appropriately control user-requested events. Nodes may also be used to maintain video calls using user-selected cameras and microphones. Furthermore, it can be used as a dedicated contact to provide a private experience to selected users or groups of users. Nodes can be placed in various locations within the venue, such as on stage, backstage, dressing rooms, concert hall, audience seating, and mezzanine, and may be used for additional private video calls, providing users with additional locations such as "on stage," "backstage," or throughout the venue. Video calls can also be used to capture users and serve as additional data points represented within the venue. For example, images of a user's face or other data from a user's camera can be shared with other users and used as data points to be replayed in interesting ways (e.g., how fans are reacting to the concert and capturing fan reactions).

[0016] The server may also provide ways for users to participate in the event throughout the live streaming event. Users may be prompted to access and log in to the server via their personal devices. Once access is granted, a user interface will be displayed, allowing users to participate in events taking place on the live stream / venue. These events may include, but are not limited to, drawing animations across the entire venue screen, tapping to the beat, conducting live surveys, answering trivia questions, recording user input, rendering user input on-site, rotating spotlights, following leaders, and interacting with streaming artists.

[0017] The embodiments disclosed herein generally provide a novel experience in which users interface with live performances from a distance, even when they are far from the location where the live performance is taking place. For example, users may be able to interact with and participate in events with their favorite artists like never before. Using the embodiments described herein, a venue or studio can be transformed into a virtual-enabled space in which remote users can interact with the environment. This creates an exciting and dynamic environment that is constantly evolving and becoming something new for remote users. By bringing people together in online events where users can instantly connect with artists and other users, a completely unique scenario and experience is created.

[0018] Figure 1 shows a system 100 for creating and managing a virtual-enabled studio or live performance according to one embodiment. The system 100 generally includes at least one server (hereinafter "server") 102 operably coupled to a plurality of computing devices (or clients) 104a-104c. The computing devices 104a-104n (or computing device 104) may include any one of laptops, desktop computers, mobile devices (e.g., mobile phones, tablets), etc., under the control of various users. It is also recognized that one or more of the computing devices 104a-104b may be located in a vehicle 105 that displays the live performance on a vehicle display. The vehicle 105 is a self-contained vehicle and may include a large display that allows passengers to capture the live performance from a distance from the venue 107 where the live or studio performance is performed, for example, by musicians. Similarly, by placing one or more computing devices 104a-104b in a living room or other facility of a residence, a smaller group of people may be able to view the live performance via a larger display or screen. System 100 also includes several nodes 106a to 106n located in venue 107. It is recognized that the live performances described herein may also be suitable for musicals, theatrical events, etc. For example, node 106a may correspond to at least one camera controller 108a (hereinafter referred to as camera controller 108a) that controls various cameras 110a within venue 107. For another example, node 106b may correspond to at least one lighting controller 108b (hereinafter referred to as lighting controller 108b) that controls various lights 110b within venue 107. For yet another example, node 106c may correspond to at least one robot (or prop) controller 108c (hereinafter referred to as robot controller 106c) that controls various mechanically moving props or other devices during a live or studio performance.

[0019] Users and their various computing devices 104a-104c may be located far from the venue 107 and may control various aspects of the live or studio performance while the musicians perform at the venue 107. It is understood that different users can input commands via a user interface (not shown) located on their various computing devices 104a-104c to control any one or more of the nodes 110a-110n and their corresponding cameras 110a, lighting 110b, robots 110c, etc., installed at the venue 107. As a result, the live or studio performance provides a customized performance format based on the user's preferences. For example, through an online portal such as Twitch®, users can watch broadcasts of live-streamed performances (or videos of pre-recorded performances). In one example, a user interface and data communication protocol (e.g., a live chat box) can be created to allow users to send messages while watching the live performance on their respective computing devices 104a-104c. Furthermore, one or more encoders located in venue 107 encode video and audio and send such encoded video and audio to a server. Next, a cloud database (or hosting database) (or cloud, hosting controller, or streaming platform) sends or streams the encoded video and audio to computing devices 104a-104c. Specifically, a user can enter one or more commands via a user interface located in one or more of the computing devices 104a-104c, and the commands are sent to server 102. Server 102 then sends the commands to the desired nodes 106a-106c, which perform the desired actions while a live or studio performance is taking place.Various nodes 110a-110n can control one or more of the following during a live performance: stage lighting, microphone equipment, amplification, video recording, projection, pyrotechnics, confetti cannons, robots, audio clips played in the venue, etc.

[0020] In embodiments disclosed in relation to System 100, computing devices 104a-140c may also be provided to send commands to server 102 and subsequently to nodes 106a-106n in response to keywords entered into a live chat box (e.g., Internet Relay Chat (IRC)) via an online portal presented on computing devices 104a-104c. Server 102 and / or nodes 106a-106n can convert the commands received by computing devices 104a-104c into DMX, MaxMSP, High-Definition Multimedia Interface (HDMI®), serial, etc., to trigger events during live performances at venue 107.

[0021] Figure 2 shows a method 120 for controlling one or more cameras 110a for a virtual-enabled studio or live performance, according to one embodiment.

[0022] In operation 122, the computing device 104 receives one or more commands from a user (e.g., an audience member) watching the performance. One or more of the commands may correspond to a requested movement of camera 110a, providing the user with a desired view of the performance. In another example, camera 110a (e.g., a GoPro® camera) may be positioned on one or more members of a band performing live (e.g., their heads / chests) or on their respective instruments. One or more commands may correspond to activating one or more cameras 110a positioned on any one or more of the live band members or on one or more of their instruments. The computing device 104 sends one or more commands to server 102.

[0023] In operation 124, server 102 sends one or more commands to node 110a (or camera controller 108a) in venue 107. The camera controller 108a is physically located near venue 107 where the live performance is taking place.

[0024] In operation 126, the camera controller 108a controls one or more cameras 110a in the venue to move (or rotate) to a desired camera angle or height and provide a live video stream of the performance in accordance with one or more commands sent by the user. As described above, the camera controller 108a may also selectively start / stop any cameras positioned for band members or their instruments based on commands received from the computing device 104. It is understood that the server 102 provides a detailed list or mapping of the locations of cameras 110a distributed within the venue 107 to capture the live performance, and / or the locations of cameras 110a positioned for any one or more band members or one or more of their respective instruments. The server 102 provides this mapping (or camera map) to the computing device 104, which in turn enables the user to select any number of cameras and control their operation. It is understood that the computing device 104 may also provide mappings of any of the functions disclosed herein. The user can control one or more cameras 110a to capture images of the live performance at any angle requested by the user. For example, the user can control the camera closest to the singer to zoom in on the singer during a live performance. Similarly, if the user is a guitarist and is interested in obtaining a close-up shot or view (e.g., a zoomed-in view) of the guitarist playing a guitar solo on stage, the user may control the camera 110a closest to the guitarist (or on the guitarist or on the guitar itself) to zoom in on the guitarist's fretboard and get a close-up view of the guitarist playing the solo. In addition, other cameras 110a can be placed around the venue to capture images of the entire band. The user can command such cameras 110a to zoom in or zoom out to capture close-ups of the entire band performing.Similarly, any one or more cameras 110a (or omnidirectional cameras) can provide a 360-degree view (e.g., a bird's-eye view) of the live performance if requested by the user. The camera controller 108 is known to be able to transmit any number of video streams. For example, the camera controller 108 can transmit video streams of each musician performing in venue 107 to the computing device 104. In this regard, the computing device 104 can also allow the user to select which of the video streams to display.

[0025] In operation 128, the camera controller 108a may transmit the captured live performance image to the server 102 according to a desired angle or zoom-in or zoom-out shot, as initially determined in operation 122. The server 102 then sends the captured image back to the computing device 104 for display to the user.

[0026] Figure 3 shows a method 130 for controlling lighting 110b in a virtual-enabled studio or for live performance, according to one embodiment.

[0027] In operation 132, computing device 104 receives one or more commands from a user (e.g., an audience) watching the performance. One or more of the commands may correspond to a requested movement of the lighting, providing the user with the desired lighting for the performance. Computing device 104 sends one or more commands to server 102.

[0028] In operation 134, server 102 sends one or more commands to node 110b (or lighting controller 108b) in venue 107. The lighting controller 108b is physically located near venue 107 where the live performance is taking place.

[0029] In operation 136, the lighting controller 108b controls any lighting in venue 107, such as spotlights, strobes, lighting patterns, audience lighting in venue 107, stage colors, animations, etc., in a desired manner during (or in real time) the live performance. Specifically, the lighting controller 108b converts messages (or commands) received from computing device 104 via server 102 into digital multiplex (DMX) communication (or other appropriate customized communication protocol) to control the aforementioned lighting devices and operations.

[0030] Server 102 is recognized as providing a detailed list or mapping of all lighting 110b distributed throughout the venue 107 to capture the live performance. Server 102 provides this mapping (or lighting map) to computing device 104, thereby enabling the user to select and control the operation of various lights (or lighting). It is recognized that computing device 104 can provide the detailed list or mapping instead.

[0031] In operation 138, the lighting controller 108b controls the lighting 110b as appropriate, and the camera 110a transmits captured images of the lighting 110b controlled in the live performance venue 107 via the camera controller 108a, according to the desired lighting as initially determined to the server 102 in operation 132. The server 102 then sends the captured images back to the computing device 104 for display to the user.

[0032] Figure 4 shows a method 140 for controlling one or more props (or robots 110c) for a virtual-enabled studio or live performance, according to one embodiment.

[0033] In operation 142, computing device 104 receives one or more commands from a user (e.g., an audience) watching the performance. One or more of these commands may correspond to a requested movement of robotics (or props) and provide the user with the desired prop(s) to operate during the performance. Computing device 104 sends one or more commands to server 102. In one example, the user can control any one or more of the props (or robots 110c) on stage by inputting commands to one or more of the computing devices 104a, 104b, and 104c, which can control robot node 106c. These props are electrically controlled and require mechanical movement or operation in a desired manner during (or in real time) the live performance.

[0034] In operation 144, the server 102 sends one or more commands to node 108c (or robot controller 106c) in venue 107. The robot controller 106c is located physically close to venue 107 where the live performance is taking place.

[0035] In operation 146, the robot controller 106c may activate / deactivate a desired prop 110c in the venue according to one or more commands received from the computing device 104 via the server 102. The node 106c translates the message received from the server 102 into a communication protocol (or other appropriate customized communication protocol) that controls the operation of the aforementioned robot / prop. The prop may correspond to a mechanical device (or robot) placed around or on the stage that an artist may use to enhance the user's concert experience. For example, consider the heavy metal band Iron Maiden®, such a band has a mascot known as "Eddie" or "Eddie the Head," in which a large mechanical robot in the shape of Eddie is assembled on stage. The robot resembling Eddie is known to appear on stage with the band and move around the stage while the band plays various songs. In this case, the user can choose to control Eddie's various movements via commands entered into computing devices 104a-104c and sent to the robot control device 106c via server 102. In this example, node 106c can convert the commands received by server 102 into serial data to control Eddie's movements on stage during a live performance.

[0036] It is understood that server 102 provides the user with a detailed list or mapping of all props 110c distributed throughout venue 107 that can be controlled during a live performance. Server 102 provides this mapping (or prop map) to computing device 104, thereby enabling the user to select different props and control their behavior. It is also understood that computing device 104 can provide a detailed list of mappings for all props 110c.

[0037] In operation 148, the robot control device 106c controls the props as appropriate, and the camera 110a sends captured images of the modified props to the server 102 via the camera control device 108a. The server 102 then sends the captured images back to the computing device 104 for display to the user.

[0038] Figure 5 shows a method 150 for controlling activities for a virtual-enabled studio or live performance, according to one embodiment.

[0039] In operation 152, computing device 104 receives one or more commands from a user (e.g., an audience) watching the performance. One or more of these commands may correspond to controlling various items (or items 110n) on stage, such as pyrotechnics, confetti cannons, video / audio projection onto a screen, microphone equipment, and amplifiers, in a desired manner during (or in real time) the live performance. Computing device 104 sends one or more commands to server 102. For example, a user may input commands to one or more of computing devices 104a, 104b, and 104c, which may control controllers 106n (e.g., any number of displays in venue 107, such as pyrotechnic nodes, confetti cannon nodes, video / audio projectors, televisions, LED panels (or LED walls)), microphone equipment nodes (e.g., microphones such as binaural microphones, amplifiers, etc.) in a desired manner during (or in real time) the live performance. The user can control various aspects (or audio properties), such as changing the tone of a guitar or bass, or increasing the volume of a specific instrument. It is understood that System 100 can automatically increase the volume of any given instrument in response to the user selecting a dedicated video stream for the musician playing the instrument. Similarly, the user can create their own music mix based on the audio received from Venue 107, adding their personal audio preferences such as equalization, effects, and compression. Microphones, such as binaural microphones, may be placed in Venue 107 (for example, scattered among the audience of Venue 107), so that the microphones can capture the atmosphere and feel of the audience in Venue 107, and the captured atmosphere can be transmitted by the microphones to the computing device 104 via Server 102 and various controllers 106n.The computing device 104 can also send commands to selectively activate and deactivate one or more microphones in venue 107. It is recognized that the microphones may support binaural, beamforming, directional, XY, French National Broadcasting Corporation (ORTF) microphone equipment / recording solutions, or other appropriate technologies.

[0040] In operation 154, server 102 sends one or more commands at venue 107 to node 108n (or collectively called controllers 106n, such as pyrotechnics controller, confetti cannon controller, video / audio projection controller, microphone device controller, and amplification controller). The controllers 106c are located physically close to venue 107 where the live performance is taking place.

[0041] In operation 156, the controllers 106n may activate / deactivate items 110n (e.g., pyrotechnics, confetti cannons, video / audio projection, microphone equipment (or microphones (e.g., binaural microphones)), amplifiers, etc.) in accordance with one or more commands received from the computing device 104 via the server 102. For example, the controllers 106n can activate pyrotechnics, confetti controllers, video / audio projection, microphone equipment, and amplifiers. With respect to microphone equipment, the controllers 106n can increase or decrease the level of the microphone equipment with respect to the audio captured on stage. For example, the controllers 106b can control the audio level, and in particular, the level of a particular instrument captured by one or more microphones, to correspond to a desired amount requested by the user. Similarly, the user can adjust the amount of amplification applied to any instrument being played in the venue 107. The user can also activate any video or audio projection on any screen or monitor in the venue. The controllers 106n may also selectively activate / deactivate binaural microphones placed in the venue 107.

[0042] Server 102 is recognized to provide the user with a detailed list or mapping of all items 110n that can be controlled during a live performance. Server 102 provides this mapping (or maps) to computing device 104, which in turn enables the user to select and control various items (e.g., types of audio and / or video that can be started or stopped, confetti cannons, pyrotechnics, microphone devices for musical instruments, binaural microphones, and amplification for musical instruments).

[0043] In operation 158, the controllers 108n control the items as appropriate, and the camera 110a transmits captured images of the items 106n, which have been modified based on the command, to the server 102 via the camera controller 108a. The server 102 then sends the captured images back to the computing device 104 for display to the user. It is recognized that a soundboard is placed in the venue 107 and can transmit an audio stream wirelessly to the server 102. The server 102 then transmits the audio stream to the computing device 104. Therefore, any changes made to the microphone equipment and / or amplification are captured in the transmitted audio stream sent to the computing device.

[0044] Furthermore, it is understood that users of computing devices 104a-104c may exchange currency to obtain credits that enable them to control various nodes 106a-106n and achieve desired events that occur during the live performance. In this regard, users may use each computing device 104a-104c for any one or more of the following: control and / or access to HiFi audio provided by binaural microphones placed both within the audience and on stage during the live performance; audio solos of specific instruments of band members during the live performance; additional video streams; opportunities to interact with the artists; sharing usernames and emoticons on a projection screen placed behind the band; and special events that take place during the live performance.

[0045] Server 102 can grant different levels of prestige based on the number of credits purchased or through some other arrangement. Virtual currency allows users to pay for access and control various features (e.g., camera 110a, lighting 110b, robot 110c, items 110n) based on their prestige and virtual currency balance. A higher prestige setting associated with a user can provide such a user with higher priority, which overrides potentially conflicting commands. For example, user 1 may be considered a "base player" or customer, and user 2 may be considered a "premium player". If user 1 sends a command to control the movement of robot 110c to move forward, and user 2 sends a command to control the movement of robot 110d to move backward, server 102 determines the respective prestige levels of user 1 and user 2, and since user 2's prestige level is higher than user 1's, it triggers the command to move robot 110c backward (e.g., the command from user 2). Furthermore, if two users share similar prestige levels, server 102 can ensure that desired events in the live performance at venue 107 are achieved based on the order in which commands are received in relation to other commands. In this case, server 102 can use a time delay to enable the desired events to occur during the live performance at venue 107 once the events are started or stopped. Once the delay has expired, server 102 can process the next command to enable the start or stop of the desired events during the live performance at venue 107. In addition to executing commands for users based on prestige or status, system 100 can instead monitor or aggregate a predetermined number of commands and execute such commands based on a simple majority in terms of primary requests by users.

[0046] Figure 6 shows a method 160 for determining fame between a first user and a second user when conflicting commands are provided to control aspects related to a virtual-enabled studio or live performance, according to an embodiment.

[0047] In operation 162, server 102 receives first and second commands from first and second computing devices 104a and 140b, respectively. In operation 164, server 102 determines whether the first and second commands involve conflicting actions to be performed in venue 107. For example, server 102 may determine that the first command indicates a first lighting sequence that is different from a second lighting sequence requested via the second command. If no conflicting commands are received, method 160 proceeds to operation 168, and server 102 may execute the two commands based on the order in which the commands were received. If server 102 determines that there are conflicting commands, method 160 proceeds to operation 166.

[0048] In operation 166, server 102 evaluates or determines whether the first user and the second user have the same level of fame. For example, if the first user and the second user have the same level of fame, it is undesirable to perform conflicting changes simultaneously in venue 107, so server 102 needs to evaluate other criteria to determine whether to execute the first command or the second command. If server 102 determines that the first user and the second user have the same level of fame, method 160 proceeds to operation 168, where the commands are executed based on the order in which they were received. If server 102 determines that the first user and the second user do not have the same level of fame, method 160 proceeds to operation 170.

[0049] In operation 170, server 102 sends the first command received between the first and second commands to the venue (or one of nodes 106a to 106n), resulting in the command belonging to the user with the highest reputation level being executed first. Once the command belonging to the user with the highest reputation level is executed at venue 107, server 102 sends the next received command belonging to a user with a lower reputation level, resulting in this command being executed at venue 107. In operation 170, server 102 sends the command belonging to the user with the highest reputation level to venue 107, thereby executing this command.

[0050] A chatbox (or other user interface medium) can be used in computing devices 104a-104c to process messages, interpret IRC, and determine whether a user has access rights to system 100, and to send commands via a DAC. Commands sent from computing devices 104a-104c (e.g., IRC) can be sent (directly or indirectly) to nodes 108a-108n. This can be done via scripts, socket commands, DMX, serial, etc. A digital-to-analog converter (DAC) may optionally be used to send voltages or currents that trigger relays, robots, etc. It is recognized that IRC (or the chatbox) can send many types of analog or digital commands to various nodes. Examples of digital command types may include MIDI, serial, DMX lighting controllers, TCP, etc., while examples of analog signals may include lighting, robots, relays, meters, etc. It is recognized that system 100 can send signals based on either digital or analog. It is further recognized that nodes 108a-108n can be integrated into a single electronic unit configured to convert various commands received by server 102 into DMX, MaxMSP, HDMI®, serial, etc. The information in DMX, MaxMSP, HDMI®, and serial formats is then transmitted to corresponding cameras 110a, lighting 110b, robots 110c, and other items 110n, etc., so that one or more computing devices 104a-104c can control such devices in the requested manner.

[0051] In another example, server 102 can determine the results of voting proxies submitted to server 102 via the user interface (e.g., IRC) of computing devices 104a-104c. Voting may be used to determine the next song, trigger fog machines (e.g., pyrotechnics, robots), answer trivia questions, trigger other events, etc. Voting may also be interpreted as a meter or condition for triggering an event. For example, a “voting meter” could be used, where 500 people vote to change the lights of a live performance to red. When a threshold is reached (or a majority of votes are received), the lights can be controlled to turn red. Server 102 may also control remote computers (e.g., mobile phones, tablets, laptops (e.g., any internet-connected device)) located anywhere within venue 107 (e.g., on stage, front row, dressing room, backstage, front of the venue) and host special video calls with users to “bring users on stage.”

[0052] In another example, server 102 can enable remotely hosted group events that users can log into remotely. Furthermore, computing devices 104a-104c provide dedicated interfaces for one-time events for users to (1) press buttons, (2) vote, (3) answer trivia questions, (4) draw animations, and (5) change stage colors.

[0053] System 100 generally enables live virtual concert series that are live-streamed online. System 100 also allows fans to stream for free and provides tiered levels of access to the features and audio quality of the live performance. System 100 also offers tiered tickets for additional access to content and special merchandise.

[0054] Figure 7 shows an example of cheer credits 172 that may be issued by the system 100 of Figure 1 according to one embodiment. Figure 7 shows the various cheer credits that can be issued to a user when the user exchanges currency for credits. For example, the system 100 may allow the user to use a large projection screen behind the artist to post various fan interactions such as emojis, drag emojis onto the screen, vote on songs or deep cuts that the artist will perform, send personalized messages to the band and have them displayed on the projection screen, have one or more band members send personalized messages during the performance, and / or have some type of audio played at venue 107. Figure 7 further shows examples of currencies that can be used for various cheer credits at a certain exchange rate.

[0055] Figure 8 also shows an example of cheer credits 174 that may be issued by the system 100 of Figure 1 according to one embodiment. For example, system 100 can provide a content hierarchy with a predetermined number of cheer credits. In one example, system 100 can provide a first level (or basic level) that provides 5 cheer credits, a second level (or intermediate level) that provides 10 cheer credits, and a third level (e.g., special level) that provides 15 cheer credits. The first level may provide name posting (e.g., username of computing device) for public announcement or posting at venue 107 during a live performance and access to selected lighting patterns. The second level may provide several personalized messages and exclusive emojis or short recorded messages (sound bytes). The third level may provide different spotlights on band members and participation in one-on-one auctions.

[0056] Figure 9 shows an example of a ticket tier 176 that may be issued by the system 100 of Figure 1 according to one embodiment. Generally, the ticket tier 176 may include general admission, reserved seating, front row seating, and backstage passes. The general admission tier may offer the user advertisements, cheers to purchase or included for free, per-instance payment options, a "get the idea" option, and an opportunity to try out special features, which may result in the user purchasing higher-tier tickets. The "get the idea" option may correspond to offering a 30-second or 60-second preview of available upgrades, such as a HiFi audio experience or other appropriate features.

[0057] Reserved seating tiers may offer purchasers an ad-free feature, thereby eliminating the need for users / purchasers to be exposed to advertisements during the show. Reserved seating tiers may also offer purchased or included cheers and may offer purchasers the option to activate or deactivate more cameras than is possible in general admission tiers. Reserved seating tiers may also offer purchasers a HiFi audio experience. The HiFi audio experience may support higher quality audio, lossless codecs, and binaural processing (for example, providing the user with the perception of actually being in the audience based on how the audio reflects off the walls of venue 107).

[0058] The front row seating tier may also offer, as part of the package, an ad-free feature along with purchased or free cheers. Similarly, the front row seating tier may offer purchasers a HiFi audio experience and a near-front row seating option. The near-front row seating option generally includes a system 100 that provides a raffle where fans are randomly selected to have a one-on-one experience with band members at the concert. The front row seating tier may also offer the option to purchase headphones with head-tracking options to enhance the audio experience (e.g., the audio experience provided by JBL Quantum SPHERE 30 360®) and credits. The backstage pass tier, upon purchase, may offer, as part of the package, purchased or free cheers along with an ad-free feature. Similarly, the backstage pass tier may offer purchasers a HiFi audio experience and greater control over more cameras 110a within the venue 107 than is offered by the other tiers. The backstage pass tier may also offer users the perk of receiving a customized and signed special head-tracking system, such as signed JBL headphones. The backstage pass also includes front-row seating options and VIP lounge access. Fans who gain access through the VIP lounge features provided by System 100 can appear on a mobile device (e.g., a tablet) placed in the dressing room of Venue 107, allowing users to listen to conversations with the band members and speak with them before and after the show.

[0059] Figure 10 shows an example of a special feature 178 that may be issued by the system 100 of Figure 1 according to one embodiment. The special features 178 provided by the system 100 include the front row option and the VIP lounge option described above. The front row option allows a user to be “pulled up” onto the stage, and the front row option allows fans to pay a fee for additional cheering. Similarly, the front row option allows a user to enter their name into a raffle multiple times. Special feature 178 also offers a “one-on-one option,” in which fans with backstage pass tiers can participate in an auction for access to a one-on-one private encore with a band member at the end of the live performance.

[0060] Figure 11 shows an example of a special offer 178 that may be issued by the system of Figure 1 according to one embodiment. As described above, the special offer 178 may be compatible with headphones (or headphone systems) based on head tracking, such as JBL QuantumONE® headphones. System 100 enables the user to take advantage of head tracking in order to provide the user with a live experience.

[0061] Figure 12 shows an example of a playlist event 180 that may be issued by the system 100 of Figure 1 according to one embodiment. The playlist event 180 may only be available to events in which people attending the live performance at venue 207 can participate. For example, the playlist event 180 may include a super fan shootout, a writer, a decibel meter, and live voting. In the "super fan shootout," two or more fans are selected to answer trivia questions about the band. The fan who wins the trivia contest, for example, will be given credit or signed merchandise. The "writer event" corresponds to users participating via their respective mobile devices selecting a prompt to raise a writer on the user interface of their respective computing device. Thus, the writer is displayed on the screen. In the "decibel meter event," users tap a button or prompt on the screen (or user interface) as quickly as possible via computing device 104, and a meter located on either server 102 or a node at venue 107 can measure how quickly the fan is tapping such a button. Taps on computing device 104 are converted into cheers by computing device 104 or server 102 and played back on computing device 104. In a "live voting event," fans (or users) can vote for band members to perform specific actions. Such actions may include, forcing a band member to have a bucket of paint thrown over their head, throwing a pie in their face, jumping from the stage into the audience, and / or shaving their head.

[0062] Figure 13 shows an example of additional playlist events 182 that may be issued by the system 100 of Figure 1 according to one embodiment. For example, additional playlist events 182 may include an evolving canvas event, a red vs. blue event, an improvisation with the band event, and a spotlight event. The evolving canvas event involves live painting, one pixel at a time, performed via computing device 104 and then displayed on a projection screen in venue 107. This also includes making the brush size correspond to the ticket hierarchy. The red vs. blue event involves dividing users on each computing device 104 into two or more teams, such users competing against each other by tapping to the beat on computing device 104, competing in band trivia, or playing games. The virtual color of the stage is determined by the winning team, which is then presented and displayed on computing device 104. The improvisation with the band event involves creating an easy-to-use sequencer for fans to electronically submit scripts (e.g., musical notes) via computing device 104. The submitted manuscripts will be played sequentially by the band on stage, with the band playing along. The notes may be divided by quantization and may even be in the pentatonic scale.

[0063] Figure 14 shows an exemplary user interface 190 on the computing device 104 of the system 100 of Figure 1, according to one embodiment. The user interface 190 may provide an upgrade ticket field 192, a merchandise purchase field 194, a cheer and bid field 196, and a dialog box 198. Users can upgrade their ticket tier by selecting the upgrade ticket field 192 and upgrading to either a reserved seat tier, a front row tier, or a backstage pass, as commonly shown in Figure 9 176. Users can purchase merchandise associated with the band through the merchandise field 194. Users can also control the nature of the show by cheering and bidding through the cheer and bid field 196. The dialog box 198 allows users to post / comment together with other users who are virtually viewing the event. References herein to entering or selecting options via the computing device 104 are also recognized to include the transmission of such data to the server 102 and the node 106 located within the venue 107.

[0064] System and method for remotely creating audio / video mixes and masters of live audio and video streams. Current live audio and video streams for all television-based programming are pre-mixed and edited by professionals. This is particularly useful for items like television, where a specific experience is desired. However, even for live or pre-recorded events, or even everyday streams, where multiple audio and video sources are available, it may be desirable for users to be able to create their own experiences and access all audio streams along with all cameras and other content present in the live performance. This could solve the challenge of pre-packaging all video and audio streams and transforming them into an experience tailored to individual preferences each time.

[0065] The embodiments disclosed herein generally enable users to access an event and create / edit different content to create their own unique experiences. Users operating computing devices may access multiple raw content streams (e.g., audio and video streams) from a live or pre-recorded event. It is recognized that content streams can extend beyond audio and video. A server may provide content streams to remote end users (e.g., computing devices (or clients)) via existing platforms such as, but not limited to, YouTube®, Twitch®, Vimeo®, and Spotify®, and by accessing these streams via computing devices or clients. The platform may also include one or more encoders located at venue 207 that encode video and audio and transmit such encoded video and audio to a server. A cloud database (or hosting database) (or cloud, hosting, or streaming platform) then transmits or streams the encoded video and audio to the computing device(s).

[0066] The embodiments disclosed herein present the challenge of temporally aligning each content stream with the others to avoid delay. It can also be implemented by running a main instance on a server at the event location, and further running remote end-user instances that can be installed by the user and give the user access to all features. The computing device (or server) enables the user to create and mix their own experiences by editing and processing the raw streams from the event. Similarly, the user can create their own music mix based on the audio received from venue 107 and add personalized audio preferences such as equalization, effects, and compression. The server (or soundboard or videoboard) may execute commands at the venue where the live or studio performance takes place. The user may enable / disable the applied settings and which content streams are selected at various points throughout the event, which are recorded in real time and finally included in the master recording.

[0067] Users can also choose which "picture-in-picture" streams they want to see, displayed alongside their selected main stream. For example, if the event is a live-streamed concert, while a guitarist is playing a solo, the user can choose a "guitar camera stream" and an "audio stream of solo guitar only." The user can also select a "drummer stream" as a smaller "picture-in-picture," and even add some of the drummer's audio to the drummer stream. As soon as the solo ends, the user can select the main camera stream and return the audio to all instruments. This can be done in real time without any delay between content switches, and all settings and selections only affect the local instance of this software running, so they do not need to affect other users' concert experiences. The entire experience can be recorded in real time and later inserted into a master recording. Users can also look back at the audio / video mix and experience the event in the way they want. It is acknowledged that users can return at a later date to remix and master the experience for a completely unique experience.

[0068] If the server is located at the event site, it can stream multiple different streams of content, including but not limited to audio and video, directly to the user's computing device. These streams may also come directly from hardware located at the venue (soundboard, video booth, etc.), and the user has access to all content streams supported and provided to the venue. The server can also be populated with the current settings used at the streaming site (e.g., sound mix, audio / video processing settings), which are selected and formulated at that site by artists, engineers, or streamers, etc.

[0069] A computing device associated with a user can access multiple streams of different content from existing streaming sites (such as YouTube®, Vimeo®, Twitch®, Spotify®, Pandora®, Soundcloud®, and Tidal®) using an Embedded Uniform Resource Locator ("URL"), an Application Programming Interface (API), or similar, but not limited to, an Embedded Uniform Resource Locator ("URL"). In such embodiments, each content stream may be loaded into the software simultaneously. When the user selects another content, the computing device can reflect the user's command by hiding or showing the specified content stream and the previous content stream. Delays and delay offsets can be determined to synchronize each content stream and ensure there is no delay when the user switches between content streams.

[0070] Both embodiments of the software approach may allow users to trigger the event via buttons that can interact with the live event, multiple different functions, and real-time output at the event venue or location. This can be implemented by sending commands via a server located within the event, or via associated streams, and via URLs of similar but not limited to socket commands, Internet Relay Chart (IRC), chatbots, etc. The way commands are triggered may be constrained by how the trigger is set up within the event space, rather than by the computing device.

[0071] Users may have a main interface screen on their respective computing devices that displays their personal design experience. The computing devices owned by the user may also have submenus or various tabs that provide additional interfaces, allowing them to adjust settings for each content stream to create a mix of different content, and how the user creates such a mix. For example, the audio page might have a mixing board-like interface including knobs, faders, and sliders for adjusting microphones and instruments (e.g., wet / dry mix, overall gain, channel gain, mute / unmute, solo, etc.). The video page may provide video processing tools including previews of all selectable camera angles, filters, contrast, exposure, hue, saturation, etc. For example, a user might select multiple video streams overlaid with a picture-in-picture of another camera in the corner of the page via the main video page, or a split screen with two video streams. End users, except for streamers, artists, engineers, etc., can at any time revert any settings back to the current settings designed at the venue. This may be controlled to prevent a particular user from accidentally disrupting the experience. Various limitations may be imposed by the server to improve the end-user experience, regarding the amount of control users have over the stream, or the amount of changes they can make to equalization, wet / dry mix, overall gain, video color, etc.

[0072] Figure 15 shows a system 200 for remotely creating audio / video mixes and masters of live audio and video streams according to an embodiment. System 200 generally includes at least one server (hereinafter "server") 202 operably coupled to a plurality of computing devices (or clients) 204a-204c. Computing devices 204a-204n (or computing device 204) may include any one of laptops, desktop computers, mobile devices (e.g., mobile phones, tablets), etc., under the management of various users. System 200 also includes a soundboard 206 located in a venue 207 where, for example, a live or studio performance is performed by musicians.

[0073] At least one guitar 208 and drums 210 are operably coupled to the soundboard 206. It is recognized that various instruments (e.g., bass guitar, keyboard, vocal input, etc.) can be operably coupled to the soundboard 206. The soundboard 206 is generally configured to receive various tracks or streams (e.g., guitar stream, bass guitar stream, vocal stream, drum stream, keyboard stream, etc.) from various instruments 208, 210 and transmit such streams to the server 202 (e.g., via a wireless or wired direct connection).

[0074] The video board 212 is operably coupled to the server 202. Both the sound board 206 and the video board 212 can be referred to as media controllers. A 360-degree field of view (FOV) camera 214 (or omnidirectional camera) is operably coupled to the video board 212. Similarly, a point-of-view (POV) camera 216 is operably coupled to the video board 212. The POV camera 216 provides captured images (or close-up images) of the musician or performer. It is recognized that any number of cameras, along with video streams from the FOV camera 214 and the POV camera 216, can be operably coupled to the video board 212. It is also recognized that the server 202 may be located somewhere near the venue 207. The server 202 can then transmit the various audio streams received from the sound board 206 and the video streams received from the video board 212 to the computing devices 204a-204c associated with the users. Audio and video streams can be streamed from server 202 to computing devices 204a-204c via YouTube®, Vimeo®, Spotify®, etc. Generally, since the user is the source of the streams and algorithms (e.g., software and hardware) used on computing devices 204a-204c, this configuration allows system 200 to determine the delay between streams and adjust it appropriately to provide the user with a seamless, lag-free experience. Generally speaking, all audio / video streams are already synchronized with the artist / musician / performer at server 202 in the live venue 207. These time-aligned streams are then delivered to viewers via streaming platforms such as YouTube® and Vimeo®. Thus, complexity is reduced, and users do not experience lag issues.

[0075] Using computing devices 204a-204c, users can modify, enable, disable, etc., all settings currently configured on the audio and video streams received from venue 207. Furthermore, users can modify, enable, disable, etc., all settings configured on the audio and video streams at various times throughout the live performance, which is recorded in real time. Additionally, soundboard 206 and / or videoboard 212 can also save all audio settings (e.g., guitar, bass, vocal settings, etc.) in addition to all video settings (or camera settings) while the live performance is taking place, and provide such information to one or more computing devices 204a-204c via server 202. Users can adjust settings that would have been selected by the artist, sound engineer, or streamer at venue 207 during the live performance via computing devices 204a-204c. Users can also adjust and modify audio settings at venue 207 where the live performance took place. Similarly, users can adjust and modify video settings at venue 207 where the live performance took place. Users of computing devices 204a–204c can record modified or adjusted video and audio streams (with or without adjusted audio and video settings) and play back the recorded modified or adjusted video and audio streams. It is recognized that computing devices 204a–204c may continue to allow users to adjust / modify audio and video streams at any time.

[0076] As described above, computing devices 204a-204c can stream audio and video streams via YouTube®, Vimeo®, Twitch®, Spotify®, Pandora®, Soundcloud®, Tidal®, and others. This approach allows each content stream to be loaded simultaneously while a live performance is taking place in venue 207. When a user selects different media content on computing devices 204a-204c, the computing devices can reflect the user's command by hiding or showing the specified or selected content stream and the previous content stream.

[0077] The user can also select a “picture-in-picture” stream via any one or more of the computing devices 204a-204c, which the user may wish to be displayed on the display of computing device 204 alongside the main selected stream. For example, if the event is a live-streamed concert and a guitarist is playing a solo, the user can select a “guitar camera stream” and a “solo guitar only audio stream” via computing device 204. The user can also select the “picture-in-picture” option via computing device 204 to add part of the drummer’s audio to the stream as the guitarist plays along. As soon as the solo ends, the user can select the main camera stream via computing device 204 and return the audio to all instruments. This can occur in real time without delay during content switching. Furthermore, all settings and selections only affect the local instance on computing device 204 that modifies the audio and / or video streams, and therefore do not affect anyone else’s concert experience.

[0078] Each computing device 204a–204c may include a main interface screen that presents a personally designed experience. Within submenus or various tabs on the user interface of computing device 204, computing device 204 may provide additional interfaces for adjusting the settings of each different content stream (e.g., guitar stream, bass stream, drum stream, video stream, etc.), allowing for the creation of various content mixes. For example, computing device 204 may provide an audio page 250 (see Figure 16) that provides a similar interface to a mixing board. The audio page 250 typically includes knobs, faders, and sliders for adjusting microphone / instrument equalization, wet / dry mix, overall gain, channel gain, mute / unmute, solo, etc. An additional content page (or tuning page) 256 may include control switches (or knobs, sliders, etc.) for controlling the volume, balance, treble, and bass of the received audio stream. An editing field 258 allows the user to create or edit audio streams or tracks for each dedicated instrument. For example, editing field 258 displays knobs and faders that can be operated via user input, with each fader linked to a corresponding instrument (e.g., guitar, bass, vocals, drums, keyboard, etc.). Editing field 258 also allows the user to mix various tracks of audio, acting as a mixing desk and providing the user with the ability to balance the audio.

[0079] Furthermore, the computing device 204 may include a video page 252 that provides or displays multiple small previews of all camera angles available for the user to select from the computing device 204. The computing device 204 may also provide video processing tools via the video page 252, including filters, contrast, exposure, hue, saturation, etc. In addition, the user may select multiple video streams to be overlaid via the computing device 204. The computing device 204 may also provide picture-in-picture of another camera in the corner, a split screen containing two video streams, etc. At any time, the user can revert the settings via the computing device 204 to the current settings that are actually applied to the live performance by the artist, engineer, or streamer. The computing device 204 may be configured to ensure that a particular user does not inadvertently disrupt their experience. In one embodiment, it may be preferable to set limits on the number of setting changes to avoid disrupting the stream experience, as users may go overboard with many aspects of settings such as equalization, reverb, and gain. Such large changes to these settings may make the experience unenjoyable for the user. The computing device 204 may be configured to limit the amount of control over the stream provided by the server 202, or to limit amounts such as equalization, wet / dry mix, overall gain, and video tint, in order to improve usability for the end user.

[0080] The aspects disclosed in relation to System 200 include (i) controlling the audio stream and the type of broadcast on the audio stream, as well as controlling the camera angle and the live performance stream; (ii) providing a user interface on computing devices 204a-204c, including, for example, sliders and / or other switch mechanisms for several controls (e.g., level control for each instrument, equalization changes, wet / dry mix, etc.); (iii) an end-user configurable platform on computing devices 204a-204c that enables the user to mix audio and select the corresponding video stream from video board 202; and (iv) the audio engine of venue 207. (v) resetting from near to the default "front of the house" mix; (vi) selecting the desired video stream for a large number of songs in a live performance with multiple video streams; (vi) providing picture-in-picture from a live performance with other video streams; (vii) allowing users to record their own concert mix (e.g., video / audio) of a live performance and remix it later; (viii) streaming multi-channel content, multiple audio streams consisting of different streams of different instruments; and (ix) streaming multi-channel content, multiple audio streams and multiple video streams.

[0081] Figure 17 shows a method 300 for temporally aligning audio and video streams from a live performance, according to one embodiment.

[0082] In operation 302, server 202 receives live-streamed audio and video data from soundboard 206 and videoboard 212 located in venue 207 (or from a media controller). It is recognized that the video stream may include several video streams captured from various cameras 214 and 216 located in venue 207. For example, assuming a band is performing live in the venue, various cameras 214 may provide a first video stream capturing the entire band, and camera 216 may provide additional video streams (or point-of-view shots) of individual band members. Similarly, it is recognized that the audio stream may include various audio streams captured from various devices 208 and 210 located in venue.

[0083] In operation 303, server 202 transmits the live-streamed audio and video streams to a streaming platform (e.g., YouTube®, Vimeo®, Twitch®, Pandora®, Soundcloud®, Tidal®, etc.). This embodiment may involve encoding the video and audio to the server, and the server providing the encoded video and audio to another streaming provider, which then provides them to computing device 204.

[0084] In operation 304, each computing device 204 determines the delay between the live audio stream and the video stream (for example, all video streams provided by multiple cameras 214 and 216). In operation 306, after the delay has been calculated and is known, computing device 204 time-aligns / shifts (or synchronizes) the live audio stream and the live video stream with each other. For example, once computing device 204 has determined the delay (or playback offset rate) of all video streams, computing device 240 adjusts the video stream and audio stream based on the playback offset rate or delay to time-align the streams.

[0085] In operation 308, the computing device 204 can modify the audio and video properties of synchronized audio and video streams at the user's request. Any changes made by the user to an audio stream may correspond to changes in its audio properties. Similarly, changes made to the video stream(s) may correspond to changes in its video properties. For example, the user can selectively modify a single audio stream containing a single mix of all the audio provided by the band at venue 207 via the computing device 204. Alternatively, the user can selectively modify a single audio stream related to, for example, a guitar track provided by the band's guitarist at venue 207 via the computing device 204. The computing device 204 can enable the user to select various audio and video tracks. If the user wishes to view an aggregated video stream of the entire band, the computing device 204 can hide the remaining video streams of individual band members until the user selects them to view. Similarly, if the user desires to hear the entire mix of instruments played by the band, the computing device 204 can mute individual tracks, such as guitar, vocals, drums, and bass guitar, until the user selects to listen to them individually. It is recognized that any one or more audio streams or tracks can be played at any single instance in time.

[0086] Figure 18 shows a method 350 for providing a "picture-in-picture stream" for live performance according to one embodiment.

[0087] In operation 352, computing device 204 receives two or more video streams from server 202 via a streaming provider. In operation 354, computing device 204 displays a first video stream of the entire band, for example, during a live performance. As previously stated, computing device 204 receives two or more video streams from venue 207, but it is known that computing device 204 can play one of the two or more video streams. In the example presented in relation to method 350, it can be assumed that computing device 204 is simply playing a single video stream showing all the band members during a live performance.

[0088] In operation 356, computing device 204 receives a command from the user (via its user interface) to view a second video stream of a specific musician (e.g., a guitarist or vocalist) of a band performing during a live performance. In operation 358, computing device 204 plays both the first and second video streams in real time without delay between switching video content.

[0089] While exemplary embodiments have been described above, these embodiments are not intended to describe all possible forms of the invention. Rather, the language used in this specification is descriptive rather than limiting, and it should be understood that various modifications can be made without departing from the spirit and scope of the invention. In addition, features of various embodiments can be combined to form other embodiments of the invention.

Claims

1. A system for controlling the format of a virtual concert, and the apparatus is One or more controllers placed within the venue and configured to control the characteristics of the live performance at the venue based on at least one first signal, A server, and equipped with The aforementioned server, Transmitting the at least one first signal to the one or more controllers, The method involves receiving multiple second signals directly from multiple computing devices from users located far from the venue, each second signal indicating a user command that controls at least a portion of the live performance. The process of the plurality of second signals to determine and invalidate any conflicting commands among the user commands, After determining and invalidating any conflicting commands among the user commands, the at least one first signal is transmitted to one or more controllers to control the characteristics of the live performance. A system that is programmed to perform [some action].

2. The system according to claim 1, wherein the one or more controllers include a camera controller, the camera controller is configured to control one or more cameras positioned in the venue to move to a desired camera angle, and to transmit a first video stream of the live performance based on the desired camera angle to at least one of the plurality of computing devices.

3. The system according to claim 2, wherein one or more cameras are positioned directly on the performer at the venue, or directly on the performer's instrument.

4. The system according to claim 3, wherein the camera controller is further configured to activate one or more cameras positioned directly on the performer or on the instrument, and to provide a second video stream corresponding to a captured video stream of the performer or the performer's instrument.

5. The system according to claim 2, wherein the at least one computing device is further configured to display a mapping of the cameras arranged throughout the venue, to allow a user to select items on the mapping, and to control the cameras arranged in the venue.

6. A system for controlling the mode of a virtual concert, the apparatus being: One or more controllers placed within the venue and configured to control the characteristics of the live performance at the venue based on at least one first signal, A system comprising at least one computing device, The at least one computing device is Receiving a second signal indicating commands to control at least a portion of the live performance directly from a user located far from the venue, To transmit the at least one first signal to one or more controllers for controlling the characteristics of the live performance: It is programmed to do this, The one or more controllers include a camera controller, which is configured to control one or more cameras positioned in the venue to move to a desired camera angle and to transmit a first video stream of the live performance based on the desired camera angle to the at least one computing device. The system includes one or more controllers, each including a lighting controller configured to control aspects related to lighting arranged in the live performance based on the at least one first signal, and the camera controller further configured to transmit a captured video stream showing desired changes to the lighting in the live performance to the at least one computing device.

7. The system according to claim 6, wherein the lighting arranged in the live performance includes one or more of spotlights, strobe lights, stage lights, and animations.

8. The system according to claim 6, wherein the at least one computing device is further configured to display a mapping of the lighting arranged throughout the venue, to allow a user to select items on the mapping, and to control the lighting arranged in the venue.

9. The system according to claim 2, wherein the one or more controllers include a robot controller that controls at least one mechanical prop installed in the live performance based on the at least one first signal, and the camera controller is further configured to transmit a captured video stream to the at least one computing device corresponding to the movement of the at least one mechanical prop based on the at least one first signal.

10. The system according to claim 9, wherein the at least one computing device is further configured to display a mapping of the at least one mechanical prop installed in the venue, to enable a user to select an item on the mapping, and to control the at least one mechanical prop in the venue.

11. The system according to claim 1, wherein the one or more controllers include a first controller configured to control one or more first items corresponding to one or more of the following in the live performance: pyrotechnics, confetti cannons, video / audio projection onto a screen, audio microphone devices, and amplification, based on the at least one first signal.

12. The system according to claim 11, wherein the audio microphone device includes one or more binaural microphones placed in the venue to capture the atmosphere of the audience in the venue.

13. The system according to claim 11, wherein at least one of the plurality of computing devices is further configured to display a mapping of one or more of the one or more of the one or more of the one or more of the one or more of the one or more of the pyrotechnics, confetti cannons, video / audio projection onto a screen, audio microphone devices, and amplification in the live performance, based on the at least one first signal, and allowing the user to select one or more of the one or more of the one or more of the one or more of the one or more of the one or more of the one or more of the one or more of the one of the one or more of the one of the one of the multiple computing devices is further configured to display a mapping of the one or more

14. The system according to claim 1, wherein at least one of the plurality of computing devices is further configured to allow the user to select one or more of cheer credits, ticket tiers, and playlist events.

15. A system for controlling the mode of a virtual concert, the apparatus being: One or more controllers placed within the venue and configured to control the characteristics of the live performance at the venue based on at least one first signal, A system comprising at least one computing device, The at least one computing device is Receiving a second signal indicating commands to control at least a portion of the live performance directly from a user located far from the venue, To transmit the at least one first signal to one or more controllers for controlling the characteristics of the live performance: It is programmed to do this, The at least one computing device is further configured to allow the user to select one or more of the following: cheer credits, ticket tiers, and playlist events. The cheering credits are a system that allows for one or more of the following: a request to change the stage lighting, a vote for songs to be played at the venue, a request to send a personalized message to a performer during the live performance, and a request to provide a personalized message to a performer.

16. A system for controlling the mode of a virtual concert, the apparatus being: One or more controllers placed within the venue and configured to control the characteristics of the live performance at the venue based on at least one first signal, A system comprising at least one computing device, The at least one computing device is Receiving a second signal indicating commands to control at least a portion of the live performance directly from a user located far from the venue, To transmit the at least one first signal to one or more controllers for controlling the characteristics of the live performance: It is programmed to do this, The at least one computing device is further configured to allow the user to select one or more of the following: cheer credits, ticket tiers, and playlist events. The aforementioned ticket hierarchy is, A general admission tier for remotely viewing the live performance on at least one computing device, A reserved seating tier that allows the user to obtain HiFi audio quality while being located far from the venue. A front row seating tier that blocks the display of advertisements while viewing the live performance on the at least one computing device, and Backstage pass tiers that offer signed items from performers on stage, A system that corresponds to any one or more of the following.

17. A system for controlling the mode of a virtual concert, the apparatus being: One or more controllers placed within the venue and configured to control the characteristics of the live performance at the venue based on at least one first signal, A system comprising at least one computing device, The at least one computing device is Receiving a second signal indicating commands to control at least a portion of the live performance directly from a user located far from the venue, To transmit the at least one first signal to one or more controllers for controlling the characteristics of the live performance: It is programmed to do this, The at least one computing device is further configured to allow the user to select one or more of the following: cheer credits, ticket tiers, and playlist events. The aforementioned playlist event is, A super fun shootout event that allows users of multiple computing devices to compete against each other. A light event that allows the user of at least one computing device to select an option on each computing device to trigger a light representing the user holding a lighter during the live performance, A decibel meter event that allows the user of at least one computing device to tap an interface placed thereon within a predetermined time to simulate a fan repeatedly tapping during a live performance to increase the decibel level brought about by the audience, and A live voting event that allows the aforementioned user to vote for a performer to perform a predetermined action. A system that corresponds to any one or more of the following.

18. The system according to claim 1, wherein the server is configured to determine user reputation among multiple users of the multiple computing devices.

19. A method for controlling the manner of a virtual concert, wherein the method is Controlling the characteristics of the live performance at the venue based on at least one first signal via one or more controllers placed within the venue, Transmitting the at least one first signal to one or more controllers via the server, The server receives multiple second signals directly from multiple computing devices from users located far from the venue, each second signal indicating a user command that controls at least a portion of the live performance. The process of the plurality of second signals to determine and invalidate any conflicting commands among the user commands, After determining and invalidating any conflicting commands among the user commands, the at least one first signal is transmitted to one or more controllers to control the characteristics of the live performance. Methods that include...

20. A non-temporary computer-readable recording medium on which a computer program for controlling the manner of a virtual concert is recorded, wherein the computer program is: A command for controlling the characteristics of a live performance at the venue based on at least one first signal, via one or more controllers located within the venue, A command to transmit the at least one first signal to one or more controllers via the server, The server provides an instruction for receiving multiple second signals directly from multiple computing devices from users located far from the venue, wherein each second signal indicates a user command that controls at least a portion of the live performance. An instruction for processing the plurality of second signals to determine and invalidate conflicting commands among the user commands, After determining and invalidating any conflicting commands among the user commands, a command to send the at least one first signal to one or more controllers to control the characteristics of the live performance. Non-temporary computer-readable recording media, including [specific type of media].

Images