SYSTEM AND METHOD FOR PROVIDING RAPID CONTENT SWITCHING FOR MEDIA ASSETS CHARACTERIZED BY MULTIPLE CONTENT STREAMS DISTRIBUTED OVER A COMPUTER NETWORK - Patent application
The system generates a composite content stream from synchronized multiple streams to facilitate rapid and visually seamless transitions, addressing synchronization delays and achieving flicker fusion for immersive media playback.
Patent Information
- Application Number
- JP2024550121
- Authority / Receiving Office
- JP · JP
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2021-11-08
- Filing Date
- 2022-11-03
- Publication Date
- 2024-11-19
AI Technical Summary
Existing technologies face challenges in providing seamless and rapid transitions between multiple content streams due to delays in frame loading and synchronization issues, leading to flicker and loss of visual coherence during user-controlled switching in media playback.
A system that generates a composite content stream based on multiple synchronized content streams, allowing for smooth transitions by scaling and displaying selected portions of frames without the need for real-time fetching or processing new streams, thereby achieving flicker fusion.
Enables rapid and visually seamless switching between different angles or views of a media asset, maintaining synchronization and reducing delays to create a smooth, immersive viewing experience.
Smart Images

Figure 2024543235000001_ABST
Abstract
Description
[Technical field]
[0001] CROSS-REFERENCE TO RELATED APPLICATIONS
[0001] This application claims the benefit of priority to U.S. Provisional Patent Application No. 63 / 276,971, filed November 8, 2021. The contents of the aforementioned application are incorporated herein by reference in their entirety. [Background technology]
[0002] background
[0002] In recent years, users have come to access content on many devices and across many platforms. Moreover, the ways in which users interact with and access content (e.g., through mobile devices, gaming platforms, and virtual reality devices) are constantly changing, as is the content itself (e.g., from high definition content to 3D content and beyond). Accordingly, users are constantly seeking new types of content and new ways to interact with that content. Summary of the Invention [Means for solving the problem]
[0003] overview
[0003] Described herein are methods and systems for enabling interaction with and access to new types of content through rapid content switching of a media asset featuring multiple content streams. For example, in a media asset featuring multiple content streams, each content stream may represent an independent view of a scene of the media asset. During playback of the media asset, a user may only view content from one of the multiple content streams. The user may then switch between different content streams to view different angles, instances, versions, etc. of a scene. For example, a user may use a control device to change the viewing angle of a scene displayed on a screen. By moving the control device in a particular direction, the viewing angle of the scene displayed on the screen changes in the corresponding direction, thereby allowing the user to view the scene from a different angle.
[0004]
[0004] To generate a media asset, the system can use multiple content capture devices (e.g., cameras, microphones, etc.). The system can use content capture devices positioned close enough to each other to allow a substantially visually smooth transition when transitioning between content streams. Accordingly, in response to a user request to change a content stream (e.g., a viewing angle, an instance, a version, etc.) to a new content stream (e.g., via a user input indicating a change along vertical and horizontal axes or a change in any of the six degrees of freedom using a control device such as a joystick, mouse, or screen swipe), the system can select a content stream that can provide the appearance of a smooth transition from one content stream to the next in the presentation of the media asset.
[0005]
[0005] Accordingly, the media asset presents a seamless transition from a first content stream (e.g., a scene from one angle) to a second content stream (e.g., a scene from a second angle) from the viewpoint of the viewing user as if the user were walking around the scene. To achieve such feat, the system synchronizes each of the multiple content streams during playback. For example, when individual content streams (e.g., footage) are shot by multiple content capture devices located in close proximity (e.g., in any number of spatial arrangements, such as a circle), the system can achieve a "bullet time" effect in which a single content stream appears to rotate smoothly around an object, which effect can be achieved under user control. As a result, the played-back footage creates the illusion of a camera smoothly sweeping around the scene, and the user can view the action from any angle, including above or below the actor, or wherever the content capture device is located. During user-controlled playback, each of the independent content streams can be viewed separately in real time based on the user's selection of the independent content stream.
[0006]
[0006] However, providing media assets that enable such rapid transitions creates many technical hurdles. For example, switching between videos under user control using conventional methods and / or conventional video streaming protocols involves (i) accepting a user-initiated signal to switch videos to a server or other video playback system, (ii) storing the frame number (N) of the current frame in memory in response to the signal, (iii) opening the next video in the sequence, (iv) accessing frame N+1 of the new video and closing the previous video stream, (v) starting streaming the video to the user's device, and (vi) starting the new video at frame N+1. Due to the number of steps and the inherent need to pass information back and forth, the system cannot locate, load, and generate the new video quickly enough to provide a seamless transition. For example, when used with current software protocols, whether browser-based or standalone video players, it is not possible to open and close multiple videos at a rate that achieves flicker fusion (approximately 20-30 videos per second). Even if the connection to the server or hard drive is very fast, delays in open / close / jump to frame sequences can result in dropped frames and loss of synchronization, creating an undesirable visual effect.
[0007]
[0007] To overcome these technical hurdles and to enable smooth transitions between content streams (e.g., to achieve flicker fusion) and maintain synchronization, systems can transmit multiple content streams in parallel and generate (but not display) multiple content streams simultaneously. Unfortunately, this approach is not without technical challenges. For example, transmitting and / or generating multiple content streams simultaneously can create inherent bottlenecks in transmission speeds, whether using Internet Protocol, WI-FI, or delivery from a local drive. For example, traditional streaming video technologies are designed to deliver flicker-free video over cable, WI-FI, or locally stored files, but it is not possible to quickly and smoothly switch between many independent videos in a streaming or local environment.
[0008]
[0008] Accordingly, to overcome these technical challenges, a system generates a composite content stream based on multiple content streams for a media asset, each of the multiple content streams corresponding to a respective view of the media asset. Specifically, each of the multiple content streams has a portion dedicated to one of the multiple content streams. The system then selects which content stream (e.g., which portion of a frame of the composite stream) to generate for display based on the respective view corresponding to each of the multiple content streams. Some views (e.g., portions of a frame of the composite stream) are hidden from view, while other views are hidden from view. For example, the system can scale the selected view to fit the contours of the user interface in which the media asset is displayed (e.g., from a 1920x1080 pixel version corresponding to a portion of a frame of the composite stream to a 3840x2160 pixel version). When a new view is selected, the system simply scales the corresponding portion of the frame of the composite stream. Because there is no need to fetch (e.g., from a remote source), load, and process new streams, the system can seamlessly transition between views (e.g., achieve flicker fusion).
[0009]
[0009] In some aspects, a method and system for providing rapid content switching of a media asset featuring multiple content streams distributed over a computer network is described. For example, the system can receive a first composite content stream based on a first composite frame and a second composite frame, the first composite frame based on a first frame set, the first frame set including a first frame from each of the first multiple content streams corresponding to a first time mark of each of the first multiple content streams, the second composite frame based on a second frame set, the second frame set including a second frame from each of the first multiple content streams corresponding to a second time mark of each of the first multiple content streams, the first multiple content streams being for a media asset, and each content stream of the first multiple content streams corresponding to a respective view of a scene of the media asset. The system can then process the first composite content stream for display in a first user interface of a user device.
[0010]
[0010] Various other aspects, features and advantages of the present invention will become apparent through the detailed description of the present invention and the accompanying drawings. It should also be understood that both the general description above and the detailed description below are examples and are not intended to limit the scope of the present invention. As used in this specification and claims, the singular forms "a", "an" and "the" include plural references unless the context clearly dictates otherwise. In addition, as used in this specification and claims, the term "or" means "and / or" unless the context clearly dictates otherwise. In addition, as used in this specification, "portion" refers to a part or the whole (i.e., the whole portion) of a given item (e.g., data) unless the context clearly dictates otherwise. [Brief description of the drawings]
[0011] BRIEF DESCRIPTION OF THE DRAWINGS [Figure 1]1 illustrates an exemplary user interface for presenting a media asset through rapid content switching between multiple content streams, according to one or more embodiments. [Diagram 2]
[0012] 1 illustrates an example system for generating a media asset featuring multiple content streams, according to one or more embodiments. [Diagram 3]
[0013] 1 is another exemplary system for generating a media asset featuring multiple content streams, according to one or more embodiments. [Figure 4]
[0014] 1 is an exemplary system architecture for providing rapid content switching of a media asset featuring multiple content streams delivered over a computer network, according to one or more embodiments. [Diagram 5]
[0015] 1 is an illustration of a composite frame based on multiple content streams, according to one or more embodiments. [Figure 6]
[0016] 1 is an illustration of a concatenated composite frame based on multiple content streams, according to one or more embodiments. [Figure 7]
[0017] 1 is an illustration of a pair of concatenated composite frames based on multiple content streams in accordance with one or more embodiments. [Figure 8]
[0018] 1 is an illustration of selecting an area of a frame to zoom in, according to one or more embodiments. [Figure 9A]
[0019] 1 is an illustration of determining a sequence of views to transition to when switching between views, according to one or more embodiments. [Figure 9B] 1 is an illustration of determining a sequence of views to transition through when switching between views, according to one or more embodiments. [Figure 10]
[0020] 1 is an illustration of a playlist of a series of views according to one or more embodiments. [Figure 11]
[0021] 1 illustrates a flowchart of steps involved in providing rapid content switching for a media asset featuring multiple content streams delivered over a computer network, in accordance with one or more embodiments. [Figure 12]
[0022] 1 illustrates a flowchart of steps involved in generating a media asset featuring multiple content streams, according to one or more embodiments. [Figure 13]
[0023] 1 is an exemplary system for generating a media asset featuring multiple content streams over a large area, according to one or more embodiments. [Figure 14]
[0024] 1 illustrates an exemplary content capture device for generating a media asset featuring multiple content streams, according to one or more embodiments. [Figure 15]
[0025] 1 is an exemplary system for generating a media asset featuring multiple content streams over a large area featuring pre-selected sections having desired fields of view, according to one or more embodiments. [Figure 16]
[0026] 1 illustrates an example diagram related to zoom calculations in accordance with one or more embodiments. [Figure 17]
[0027] 1 illustrates an example diagram related to calculating tilt, according to one or more embodiments. [Figure 18]
[0028] 1 shows an example diagram relating to post-production of a media asset featuring multiple content streams, in accordance with one or more embodiments. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0012] Detailed Description of the Drawings
[0029] In the following description, for purposes of explanation, numerous specific details are set forth in order to provide a thorough understanding of the embodiments of the present invention. However, it will be understood by those skilled in the art that the embodiments of the present invention may be practiced without these specific details or with equivalent configurations. In other instances, well-known structures and devices are shown in block diagram form in order to avoid unnecessarily obscuring the embodiments of the present invention.
[0013]
[0030] 1 illustrates an exemplary user interface for presenting a media asset through rapid content switching between multiple content streams, according to one or more embodiments. For example, with rapid content switching, the media asset is presented in a seamless manner such that, from the point of view of the viewing user, it appears as if the user is walking around the scene when changing from a first content stream (e.g., a scene from one angle as captured by a first camera) to a second content stream (e.g., a scene from a second angle as captured from a second camera). It should be noted that, as described throughout this disclosure, references to viewing angles (e.g., as described in switching from one viewing angle to another) also refer to switching from one camera to another.
[0014]
[0031] For example, FIG. 1 may include a user device 100 that is currently displaying content on a user interface 102. For example, the user interface 102 may include received content for display to a user in a user interface of a web browser on the user device (e.g., the user device 100). As referred to herein, a "user interface" may include human-computer interaction and communication at a device, and may include the appearance of a display screen, a keyboard, a mouse, and a desktop. For example, a user interface may include the manner in which a user interacts with an application or a website. As referred to herein, "content" should be understood to mean electronically consumable media assets, such as television programs, pay-per-view programs, on-demand programs (similar to video-on-demand (VOD) systems), Internet content (e.g., streaming content, downloadable content, webcasts, etc.), video clips, audio, content information, pictures, rotating images, documents, playlists, websites, articles, books, e-books, blogs, advertisements, chat sessions, social media, applications, games, and / or any other media or multimedia, and / or combinations thereof. As referred to herein, the term "multimedia" should be understood to mean content that utilizes at least two of the different content forms described above (e.g., text, audio, image, video, or interactive content forms). The content can be recorded, played, displayed, or accessed by a user device, but can also be part of a live performance. The content of a media asset can be represented in a "content stream," which can be content that has an associated time element (e.g., can be played). A content stream can correspond to a stream (e.g., a series of frames played in series) to form a media asset (e.g., a video).
[0015]
[0032] In some embodiments, content can be personalized for a user based on original content and user preferences (e.g., as stored in a user profile). A user profile can be a directory of stored user settings, preferences, and information for an associated user account. For example, a user profile can include a user's installed programs and operating system settings. In some embodiments, a user profile can be a customized desktop environment or a visual representation of personal data associated with a particular user. In some embodiments, a user profile can be a digital representation of an individual's identity. User profile data can be generated based on a system that actively or passively monitors user actions.
[0016]
[0033] The user interface 102 displays the content currently being played. For example, the user can use the track bar 104 to adjust the playback of the content and perform playback actions (e.g., play, pause, or other actions). For example, the actions may relate to playing a non-linear media asset at a faster than normal playback speed or in a different order than the order in which the media asset is designed to be played, such as fast forward, rewind, skip, chapter select, segment select, segment skip, segment jump, next segment, previous segment, skip advertisement or commercial, next chapter, previous chapter, or any other action that does not play the media asset at normal playback speed. The actions may be any playback action other than "play" such that the play action plays the media asset at normal playback speed.
[0017]
[0034] In addition to normal playback operations, the system may enable a user to switch between different views of a media asset (e.g., a media asset based on multiple content streams). For example, in a media asset featuring multiple content streams, each content stream may represent an independent view of a scene of the media asset. During playback of the media asset, a user may only view content from one of the multiple content streams. The user may then switch between different content streams to view different angles, instances, versions, etc. of a scene. For example, a user may use a control device to change the viewing angle of a scene displayed on a screen. By moving the control device in a particular direction, the viewing angle of the scene displayed on a screen changes in the corresponding direction, thereby allowing the user to view the scene from a different angle.
[0018]
[0035] For example, the system may change the viewing angle displayed on the screen in response to a user input (e.g., in a particular direction) to a control device, which causes the viewing angle / orientation of the content to change in the corresponding direction. As a result, the system may make it appear to the user as if the user is moving around the scene and viewing the scene from different angles. For example, moving the joystick handle counterclockwise may cause the image to rotate clockwise or about another rotation axis relative to the screen. The user may be able to scroll from one viewing orientation to another by pressing a single button or multiple buttons (e.g., each associated with a predefined viewing angle). Additionally or alternatively, the user may choose to follow a playlist of viewing angles.
[0019]
[0036] Moreover, the system can achieve these modifications while achieving flicker fusion, which is related to the frequency at which intermittent light stimuli appear completely stable to the average human observer. Thus, the flicker fusion threshold is related to the persistence of vision. Flicker can be detected for many waveforms that represent time variations in intensity, but is traditionally most simply studied in terms of sinusoidal modulation of intensity. There are seven parameters that determine the ability to detect flicker, including the frequency of modulation, the amplitude or depth of modulation (i.e., the maximum rate of decrease in illumination intensity from its peak value), the average (or maximum) illumination intensity (these can be interconverted if the depth of modulation is known), the wavelength (or wavelength range) of illumination (this parameter and illumination intensity can be combined into a single parameter for humans or other animals where rod and cone sensitivity is known as a function of wavelength using a luminous flux function), the location on the retina where the stimulus occurs (as the distribution of photoreceptor cell types differs at different locations), the degree of light or dark adaptation (i.e., the duration and intensity of previous exposure to background light) which affects both intensity sensitivity and the temporal resolution of vision, and / or physiological factors such as age and fatigue. As described herein, the system can achieve flicker fusion according to one or more of these parameters.
[0020]
[0001] Figure 2 illustrates an exemplary system for generating a media asset featuring multiple content streams, according to one or more embodiments. For example, Figure 2 illustrates an example of a mounting matrix for surround filming configured to film a scene. In various embodiments, the surround video recording configuration 200 includes a content capture device mounting matrix 202 that is used to support and position many content capture devices 204, which can be tens, hundreds, or more content capture devices, to record the scene 206. In another embodiment, many of the content capture devices 204 are standalone content capture devices and are not mounted on a mounting matrix.
[0021] In various embodiments, user-controlled playback of multi-stream video is enabled by a recording arrangement 200, where a scene 206 is recorded simultaneously using multiple content capture devices to generate a multi-stream video, with each content capture device recording the same scene from a different direction. In some embodiments, the content capture devices can be synchronized to start recording the scene at the same time, while in other embodiments, the recorded scene can be synchronized post-synchronized on a frame number and / or time basis. In yet another embodiment, at least two of the content capture devices can record the scene consecutively. Each content capture device generates an independent content stream of the same scene, but from a different direction compared to the other content capture devices, depending on the position of the content capture device in the mounting matrix 202 or generally relative to the other content capture devices. The independently derived content streams can be tagged for identification and / or combined into one multi-stream video, allowing a user to dynamically select each of the content streams during playback for viewing.
[0022]
[0003] In some recording embodiments, the multiple content capture devices are positioned close enough to each other to allow for a substantially visually smooth transition between the content streams of the content capture devices during viewing, whether in real time or pre-recorded, when the viewer / user selects different viewing angles. For example, during playback, as the user uses a control device such as a joystick to move the viewing angle of a scene from the left to the right of the scene, the content stream changes smoothly to show the scene from the appropriate angle, as if the user himself were walking around the scene and viewing the scene from different angles. In other recording embodiments, the content capture devices are not necessarily close to each other, and the viewer / user can significantly change his / her viewing direction. In still other embodiments, the same scene can be recorded multiple times from different coordinates and / or angles in front of the same content capture devices to make it appear as if the multiple content capture devices captured the original scene from different directions. In such a configuration, each behavior can be somewhat different from similar behaviors performed at other angles to enhance the impact of that particular scene on the user. Such recordings can later be synchronized and presented to a viewer / user to create the illusion of viewing the same scene from multiple angles / directions.
[0023]
[0004] During user-controlled playback, each of the independent content streams can be viewed separately in real time or recorded for later viewing, based on the user's selection of the independent content stream. Typically, during playback, the user is unaware of the fact that the independent content streams may not have been recorded simultaneously. In various embodiments, the independent content streams can be electronically mixed to form a single composite signal for transmission and / or storage, from which the user-selected content stream can be separated by electronic techniques such as frequency filtering or other similar signal processing methods. Such signal processing techniques include both digital and analog techniques, depending on the type of signal.
[0024]
[0005] In various embodiments, multiple content streams can be combined into a multi-stream video, each of which is selectable and separable from the multi-stream video during playback. The multi-stream video can be packaged as a single video file or as multiple files that can be used together as one target video. An end user can purchase a physical medium (e.g., a disk) that contains the multi-stream video for viewing at variable angles under user control. Alternatively, a user can download, stream, or otherwise obtain the multi-stream video and view it at different viewing angles and orientations under his control. A user may be able to download or stream only the recordings of the orientations / angles he / she wants to view later.
[0025]
[0006] In various embodiments, after filming is completed, the footage from each camera or content capture device can be transferred to a computer hard drive or other similar storage device. In some embodiments, the content capture device acquires an analog content stream, and in other embodiments, the content capture device acquires a digital content stream. The analog content stream can be digitized before storage on a digital storage device such as a computer hard disk. In some embodiments, each content stream or footage can be labeled or tagged in the mounting matrix with a number or similar identifier that corresponds to the content capture device from which the content stream was obtained. Such identifiers can generally be mapped to viewing angles / orientations available to the user during viewing.
[0026]
[0007] In various embodiments, the content stream identifiers are assigned by the content capture devices themselves. In other embodiments, the identifiers are assigned by a central controller of multiple content capture devices. In yet other embodiments, the content streams can be recorded independently by each content capture device, such as a full video camera, onto separate media, such as tape, and then tagged manually or automatically during later consolidation of all the content streams into a single multi-stream video.
[0027]
[0008] In various embodiments, the mounting matrix 202 may be one-dimensional, two-dimensional, or three-dimensional, such as a curved, spherical, or flat mounting system that provides a framework for housing a matrix of content capture devices mounted on the outside (scene-facing side) of the mounting matrix, with lenses oriented inward toward the center of the curved matrix. 360° coverage around the scene may be provided by enveloping the scene with a spherical mounting matrix completely covered by cameras. As will be further described below, for large scenes, some or all of the content capture devices may be individually positioned at desired locations around the scene. In some embodiments, the mounting matrix and some of the individual content capture devices may be dynamically movable, for example by assembling them on a wheeled platform, so that the scene can be followed during active filming.
[0028]
[0009] In cases where a spherical or near-spherical mounting matrix is used to encase the target scene during capture, similar to the camera lenses discussed above, light can be delivered through a series of small holes in the mounting matrix, and due to their regular arrangement, shape and luminosity, these lights can also be easily recognized or removed in post-production.
[0029]
[0010] In various embodiments, the recording configuration 200 includes a mounting matrix 202 used to position and hold content capture devices substantially in focus on the scene 206, with different content capture devices configured to provide a 3D effect and a stronger or enhanced 3D effect, respectively.
[0030]
[0011] One of the functions of the mount matrix 202 is to provide a housing structure for cameras or other recording devices that are mounted in a predefined or regular pattern close enough to each other to facilitate smooth transitions between content streams during playback. The shape of the mount matrix modifies the user experience during playback. The ability to transform the shape of the mount matrix based on the scene being captured allows for different recording angles / orientations and therefore different playback experiences.
[0031]
[0012] In various embodiments, the mounting matrix 202 has sufficient structural rigidity to reliably and stably support many content capture devices, while being flexible enough to bend around a target scene to provide a surround effect at different viewing angles of the same target scene. In various embodiments, the mounting matrix 202 can be a substantially rectangular plane that can be bent in two different dimensions of its plane (e.g., horizontally and vertically) to surround the target scene from left and right (horizontally) or from above and below (vertically). In various other embodiments, the mounting matrix 202 can be a plane that can be configured to take on various planar shapes, such as spherical, hemispherical, or other 3D planar shapes. Different shapes of the mounting matrix allow different recording angles, and therefore different playback viewpoints and angles.
[0032]
[0013] In various embodiments, the selected pair of content capture devices and corresponding image data streams can provide various degrees of 3D visual effects. For example, a first content capture device pair can provide an image data stream that, when viewed simultaneously during playback, produces a 3D visual effect with a corresponding perspective. A second content capture device pair can provide an image data stream that, when viewed simultaneously during playback, produces a different 3D visual effect with a different and / or deeper corresponding perspective compared to the first camera pair, thus enhancing and enhancing the stereoscopic effect of the camera pair. Other visual effects can be produced using selected camera pairs that are not on the same horizontal plane, but are separated along a path in 2D or 3D space on the mount matrix. In various other embodiments, the mount matrix 202 is not used. These embodiments are further described below with reference to FIG. 3.
[0033] In some embodiments, at least one or all of the content capture devices are standalone, independent cameras, while in other embodiments, each content capture device is a networked image sensor coupled to a central recording facility. In yet other embodiments, the content capture device is a lens for collecting and transmitting light to one or more image sensors over an optical network, such as a fiber optic network. In still other embodiments, the content capture device may be a combination of one or more of the above.
[0034]
[0015] In various embodiments, the content streams generated by the content capture devices are pre-synchronized before starting the recording of a scene. Such pre-synchronization can be performed, for example, by a single remote control device sending a broadcast signal to all the content capture devices to start recording simultaneously by all the content capture devices. In other embodiments, the content capture devices can be coupled to each other to continuously synchronize the start of recording and their respective frame rates during operation. Such continuous synchronization between the content capture devices can be performed by using various techniques, such as using a broadcast running clock signal, using a digital message passing bus, and the like, depending on the complexity and functionality of the content capture devices.
[0035]
[0016] In another embodiment, at least some of the content streams generated by the content capture devices are post-synchronized after the recording of a scene. The purpose of the synchronization is to match corresponding frames of multiple content streams that recorded the same scene from different angles (but substantially simultaneously). Post-synchronization can be performed using various techniques, such as time-based techniques, frame-based techniques, content matching, and the like.
[0036]
[0017] In various embodiments, the time-based technique uses a global timestamp in each content stream and matches corresponding frames together based on their respective timestamps. The frame-based technique uses a frame count from a starting common frame position in all content streams to match subsequent frames in the content streams. For example, the starting common frame may include the first frame or frames of a special scene recorded for this purpose, such as a stripe pattern. The content matching technique may use elements of the image frame content to match corresponding frames. Those skilled in the art will appreciate that other methods for post-synchronization may be used without departing from the spirit of this disclosure.
[0037]
[0018] In various embodiments, the surround video recording configuration can be fully integrated with current 3D recording and / or viewing technology by employing an offset between content capture devices recording the same scene, positioned at a predefined distance from each other. Since the content streams from different content capture devices can be selected by the user during viewing, an enhanced or exaggerated 3D effect can be affected by selecting a content stream from a recording content capture device that is further away than the cameras used in normal 3D stereo recording, which are set at a slight distance away (usually roughly the distance between the human eyes). This dynamic selectability of content streams provides variable 3D characteristics while viewing a scene. In recent years, 3D videos and movies have become very popular, and "4D" surround video, which allows 3D videos to be dynamically viewed from various angles, further enhances this trend.
[0038]
[0019] Generally, it is not necessary to employ multiple soundtracks in a surround video recording system, and a single master soundtrack may generally suffice, but if each content capture device or camera on the mounting matrix includes an attached or built-in microphone, and the soundtrack of each content stream is switched along with the corresponding content stream, the surround sound effect of actually moving the sound to match the camera view can be achieved through a single playback speaker, as opposed to a conventional surround sound system that requires multiple speakers. For example, in a dialogue scene, a content stream is selected from a corresponding camera position that is closer to a particular actor during filming, so that the actor's voice will sound louder than a content stream that corresponds to a camera that is further away from the actor.
[0039] For example, while providing rapid content switching of a media asset featuring multiple content streams distributed over a computer network, the system can determine a particular audio track (e.g., from each content capture device) corresponding to a composite content stream. For example, the composite content stream can be based on a first composite frame and a second composite frame, the first composite frame being based on a first frame set, the first frame set including a first frame from each of the first multiple content streams corresponding to a first time mark of each of the first multiple content streams. Additionally, the second composite frame is based on a second frame set, the second frame set including a second frame from each of the first multiple content streams corresponding to a second time mark of each of the first multiple content streams.
[0040]
[0021] As the system selects frames for inclusion in the composite content stream, the system can also retrieve audio samples corresponding to the frames from each content capture device. For example, the system can determine a composite audio track for presentation with a first composite content stream. In such a case, the composite audio track can include a first audio track corresponding to the first composite frame and a second audio track corresponding to the second composite frame. Moreover, the first audio track can be captured by a content capture device that captured the first set of frames, and the second audio track can be captured by a content capture device that captured the second set of frames.
[0041]
[0022] Those skilled in the art will appreciate that the surround imaging system can be applied to still images instead of full-motion imaging. The use of still cameras in the mounting matrix allows a user to "move around" an object photographed by the system by changing the viewing angle of the object.
[0042]
[0023] In various embodiments, surround video systems can be used to address the video piracy problem. The problem media producers face is that it is very easy for viewers / users to record content and distribute it on the Internet. It is extremely difficult to pirate the multiple content streams provided by a surround video system and still provide an entire interactive viewing experience. While it may be possible to pirate a single viewing stream, record it, and distribute it, there is no easy way to access the entire set of camera angles that make up the surround video experience.
[0043]
[0024] Figure 3 is another exemplary system for generating a media asset featuring multiple content streams, according to one or more embodiments. For example, Figure 3 shows an example of a surround filming setup in which cameras configured to film a scene are independently positioned. In some embodiments, instead of using one integrated mounting matrix, independently positioned content capture devices 302, such as video cameras, are mounted on independent supports 304, such as tripods, to record a scene 306. In such an embodiment, which may be employed in filming a large area, such as an outdoor scene, the content capture devices can be positioned at any point around the target scene and record film substantially simultaneously or consecutively. Synchronization can be performed after image capture for the different content streams so obtained. If the content capture devices are separately installed, simultaneous synchronization by wired or wireless methods is also possible. Various techniques can be used to specify the various content capture device locations relative to each other, such as GPS, 3D grid-based specifications, property lines, and the like. In general, knowing the physical location and line of sight direction of the content capture devices can determine the viewing angle or direction of the target scene.
[0044]
[0025] Figure 4 is an exemplary system architecture for providing rapid content switching of a media asset featuring multiple content streams distributed over a computer network, according to one or more embodiments. For example, system 400 may represent components used to provide rapid content switching of a media asset featuring multiple content streams, as shown in Figure 1. As shown in Figure 4, system 400 may include a mobile device 422 and a user terminal 424. Although shown in Figure 4 as a smartphone and a personal computer, respectively, it should be noted that mobile device 422 and user terminal 424 may be any computing device, including, but not limited to, a laptop computer, a tablet computer, a handheld computer, other computing equipment (e.g., a server) (including "smart", wireless, wearable and / or mobile devices).
[0045] FIG. 4 also includes a cloud component 410. The cloud component 410 may alternatively be any computing device described above, including any type of mobile terminal, fixed terminal, or other device. For example, the cloud component 410 may be implemented as a cloud computing system and may include one or more component devices. It should also be noted that the system 400 is not limited to three devices. Users may, for example, utilize one or more devices to interact with each other, with one or more servers, or with other components of the system 400. It should also be noted that although one or more operations are described herein as being performed by a particular component of the system 400, the operations may be performed by other components of the system 400 in some embodiments. By way of example, although one or more operations are described herein as being performed by a component of the mobile device 422, the operations may be performed by one or more of the cloud components 410 in some embodiments. In some embodiments, the various computers and systems described herein may include one or more computing devices programmed to perform the described functions. Additionally or alternatively, multiple users may interact with system 400 and / or one or more components of system 400. For example, in one embodiment, a first user and a second user may interact with system 400 using two different components.
[0046] With respect to the components of the mobile device 422, the user terminal 424, and the cloud component 410, each of these devices can receive content and data via an input / output (hereinafter, "I / O") path. Each of these devices may also include a processor and / or control circuitry for sending and receiving commands, requests, and other suitable data using the I / O path. The control circuitry may include any suitable processing circuitry, memory circuitry, and / or input / output circuitry. Each of these devices may also include a user input interface and / or a user output interface (e.g., a display) for use in receiving and displaying data. For example, as shown in FIG. 4, both the mobile device 422 and the user terminal 424 include a display for displaying data (e.g., conversation responses, queries, and / or notifications).
[0047]
[0028] Additionally, as the mobile device 422 and the user terminal 424 are shown as touch screen smartphones, their displays also serve as user input interfaces. Note that in some embodiments, the devices may not have a user input interface or a display, but may instead use another device (e.g., a dedicated display device such as a computer screen, and / or a dedicated input device such as a remote control, mouse, voice input, etc.) to receive and display content. Additionally, the devices of the system 400 may execute applications (or other suitable programs). The applications may cause the processor and / or control circuitry to perform operations related to providing rapid content switching for media assets featuring multiple content streams.
[0048]
[0029] Each of these devices may also include electronic storage. Electronic storage may include non-transitory storage media that electronically store information. The electronic storage media of electronic storage may include one or both of (i) system storage provided integrally with the server or client device (e.g., substantially non-removable), and (ii) removable storage removably connectable to the server or client device, for example, via a port (e.g., USB port, Firewire port, etc.) or drive (e.g., disk drive, etc.). Electronic storage may include one or more of optically readable storage media (e.g., optical disks, etc.), magnetically readable storage media (e.g., magnetic tape, magnetic hard disk drive, floppy drive, etc.), charge-based storage media (e.g., EEPROM, RAM, etc.), solid-state storage media (e.g., flash drives, etc.), and / or other electronically readable storage media. Electronic storage may include one or more virtual storage resources (e.g., cloud storage, virtual private networks, and / or other virtual storage resources). The electronic storage may store software algorithms, information determined by a processor, information obtained from a server, information obtained from a client device, or other information that enables the functionality described herein.
[0049] 4 also includes communication paths 428, 430, 432. The communication paths 428, 430, 432 may include the Internet, a cellular network, a mobile voice or data network (e.g., 5G or LTE network), a cable network, a public switched telephone network, or other types of communication networks, or a combination of communication networks. The communication paths 428, 430, 432, separately or together, include one or more communication paths, such as a satellite path, an optical fiber path, a cable path, a path supporting Internet communication (e.g., IPTV), a free space connection (e.g., for broadcast signals or other wireless signals), or any other suitable wired or wireless communication path, or a combination of such paths. A computing device may include additional communication paths linking multiple hardware, software, and / or firmware components operating together. For example, a computing device may be implemented by a cloud of computing platforms operating together as a computing device.
[0050]
[0031] The cloud component 410 may also include control circuitry configured to perform various operations necessary to generate alternative content. For example, the cloud component 410 may include cloud-based storage circuitry configured to generate alternative content. The cloud component 410 may also include cloud-based control circuitry configured to perform a process for determining alternative content. The cloud component 410 may also include cloud-based input / output circuitry configured to present a media asset through rapid content switching between multiple content streams.
[0051]
[0032] The cloud component 410 may include a model 402, which may be a machine learning model (e.g., as illustrated in FIG. 4). The model 402 may take in an input 604 and provide an output 406. The input may include multiple datasets, such as a training dataset and a test dataset. Each of the multiple datasets (e.g., the input 404) may include a data subset associated with a transferable view. In some embodiments, the output 406 may be fed back to the model 402 as an input to train the model 402 (e.g., alone or in combination with a user indication of the accuracy of the output 406, a label associated with the input, or other reference feedback information). For example, the system may receive a first labeled feature input, where the first labeled feature input is labeled with a known view for transfer based on the first labeled feature input. The system may then train a first machine learning model to classify the first labeled feature input with the known view.
[0052] In another embodiment, the model 402 can update its configuration (e.g., weights, biases, or other parameters) based on an evaluation of its predictions (e.g., output 406) and reference feedback information (e.g., a user indication of accuracy, a reference label, or other information). In another embodiment where the model 402 is a neural network, the connection weights can be adjusted to reconcile the difference between the neural network's predictions and the reference feedback. In a further use case, one or more neurons (or nodes) of the neural network can request to send their respective errors back through the neural network (e.g., backpropagating the errors) to facilitate the update process. The connection weight updates can be reflective of the magnitude of the errors backpropagated after a forward pass is completed, for example. In this way, for example, the model 402 can be trained to generate better predictions.
[0053]
[0034] In some embodiments, the model 402 may include an artificial neural network. In such an embodiment, the model 402 may include an input layer and one or more hidden layers. Each neural unit of the model 402 may be connected to many other neural units of the model 402. Such connections may be compulsory or inhibitory in their effect on the activation state of the connected neural units. In some embodiments, each individual neural unit may have a summation function that combines all the values of its inputs. In some embodiments, each connection (or the neural unit itself) may have a threshold function such that a signal must exceed that threshold before it is propagated to other neural units. The model 402 is not explicitly programmed, but rather is capable of self-learning and training, and may be significantly better at certain areas of problem solving than traditional computer programs. During training, the output layer of the model 402 corresponds to the classification of the model 402, and inputs known to correspond to that classification may be input to the input layer of the model 402 during training. During testing, inputs with unknown classifications may be plugged into the input layer and a determined classification may be output.
[0054]
[0035] In some embodiments, model 402 may include multiple layers (e.g., where a signal path crosses from an earlier layer to a later layer). In some embodiments, backpropagation techniques may be utilized by model 402 where a forward stimulus is used to reset the weights of the "forward" neural units. In some embodiments, the stimuli and inhibitions of model 402 are more fluid and their connections interact in a more chaotic and complex manner. During testing, the output layer of model 402 may indicate whether a given input corresponds to a classification of model 402 (e.g., the view provides a seamless transition).
[0055] In some embodiments, the model 402 can predict a sequence of views that can be transitioned to provide a seamless transition. For example, the system can determine that certain characteristics of the views are likely to indicate the prediction. In some embodiments, the model (e.g., model 402) can automatically perform an action (e.g., select one or more views of the sequence of views) based on the output 406. In some embodiments, the model (e.g., model 402) may not perform any action. The output of the model (e.g., model 402) is used only to determine the locations and / or views to recommend.
[0056]
[0037] The system 400 also includes an API layer 450. In some embodiments, the API layer 450 can be implemented on the mobile device 422 or the user terminal 424. Alternatively or additionally, the API layer 450 can reside on one or more of the cloud components 410. The API layer 450 (which can be a REST or web service API layer) can provide a decoupled interface to the data and / or functionality of one or more applications. The API layer 450 can provide a language-agnostic, common way to interact with the applications. Web service APIs provide a well-defined contract, called WSDL, that describes the behavior of the service and the data types used to exchange information. REST APIs typically do not have this contract and are instead documented with client libraries for the most common languages, including Ruby, Java, PHP, and JavaScript. SOAP web services have traditionally been adopted by enterprises to expose internal services and to exchange information with partners in B2B transactions.
[0057]
[0038] The API layer 450 can use a variety of architectural configurations. For example, the system 400 can be based partially on the API layer 450, resulting in a strong adoption of SOAP and RESTful web services and the use of resources such as service repositories and developer portals, but with low governance, standardization, and separation of concerns. Alternatively, the system 400 can be based entirely on the API layer 450, resulting in separation of concerns between layers such as the API layer 450, between services, and between applications.
[0058]
[0039] In some embodiments, the system architecture can use a microservices approach. Such a system can use two types of tiers, a front-end tier and a back-end tier, where the microservices reside. In this type of architecture, the role of the API tier 450 can be to provide integration between the front-end and the back-end. In such a case, the API tier 450 can use RESTful APIs (exposing to the front-end or communicating between microservices). The API tier 450 can use AMQP (e.g., Kafka, RabbitMQ, etc.). The API tier 450 can use early-stage usage of new communication protocols such as gRPC, Thrift, etc.
[0059]
[0040] In some embodiments, the system architecture can use an open API approach. In such cases, the API layer 450 can use commercial or open source API platforms and their modules. The API layer 450 can use a developer portal. The API layer 450 can apply WAF and DDoS protection using strong security constraints, and the API layer 450 can use RESTful APIs as a standard for external integration.
[0060]
[0041] Figure 5 is an illustration of a composite frame based on multiple content streams, according to one or more embodiments. For example, Figure 5 may include two composite frames (e.g., frame 500 and frame 550). Frame 500 may include a frame set that may include a first frame (e.g., frame 502) from each of the first multiple content streams that corresponds to a first time mark in each of the first multiple content streams.
[0061]
[0042] Each of the frames in the frame set may be a reduced and / or compressed version of a frame. Each of the frames may also correspond to a portion of the composite frame. Moreover, these portions may be shaped to fit evenly within the bounds of the composite frame when positioned next to each other, as shown in frame 500. For example, the system may scale frame 502 (e.g., a selected view) from 1920x1080 pixels (corresponding to the portion of frame 500 that includes frame 502) to a 3840x2160 pixel version. For example, the system may increase the size and / or scale of frame 502 to fit the contours of a user interface (e.g., user interface 102 (FIG. 1)) in which the media asset is displayed. By using frames of the same size and shape in the composite frame, the system may apply the same scaling process and scale factor to reduce processing time. For example, when a new view is selected, the system simply scales the corresponding portion of the frame in the composite stream. Because there is no need to fetch a new stream (eg, from a remote source), load, and process the new stream, the system can seamlessly transition between views (eg, achieving flicker fusion).
[0062] For example, the system may apply image scaling to frame 502 to resize a digital image representing frame 502 to the size of the user interface. For example, when scaling a vector graphics image, the system may use geometric transformations to scale the graphics primitives that make up the image (e.g., frame 502) without loss of image quality. When scaling a raster graphics image, a new image may be generated that has more or fewer pixels. For example, the system may scale down an original version of frame 502 to create frame 500. Similarly, the system may scale up 502 when generating it for display.
[0063]
[0044] For example, frame 500 may include multiple content streams (1 to N), each of which is captured by a content capture device structured in a matrix (e.g., mounting matrix 202 (FIG. 2)) or by cameras (e.g., content capture device 302) adjacent to one another. In this example, eight content streams are captured using adjacent content capture devices (e.g., video cameras).
[0064]
[0045] Before being hosted on the server, local hard drive, or other component of Figure 4, the content streams can be edited and time-synchronized to be of equal duration. The content streams can then be arranged and embedded into a composite content stream (e.g., represented by frame 500). In this example, the content streams are compiled into two sets (e.g., represented by frames 500 and 550), each of which is 3840x2160 pixels in size, made up of four 1920x1080 individual content streams of equal duration (e.g., 10 seconds, 30 minutes, 2 hours).
[0065]
[0046] The two sets of frames (e.g., corresponding to frames 500 and 550) can be uploaded to a server and the system can either transfer the composite content stream to the user device or store it on the user's local computer drive.
[0066]
[0047] Before playback begins, the user interface (e.g., a web browser, a custom app, or a standalone video player) can load each composite content stream into a separate instantiation of a player and synchronize the composite content streams in time. Additionally or alternatively, the system can open additional instances of a local player and synchronize the various composite content streams. The system can balance the number of local player instances and the number of composite streams based on the resources available on the system.
[0067]
[0048] Upon initiation of playback (which may be initiated based on receipt of a command from the user or via software embedded in the user interface (e.g., a web browser, media player, or custom application)), the system may generate a content stream. During playback, the system may generate a single content stream of the composite content stream for display (and / or scale the content stream to fit the contours of the user interface). For example, in FIG. 5, the content stream (e.g., "Video 1") in the upper left of frame 500 is made visible in the user's media player or browser and playback begins.
[0068]
[0049] All content streams (and / or composite content streams) are synchronized in time, so that even if all content streams (and / or composite content streams) except for the content stream in the upper left corner of frame 500 (e.g., "Video 1") are hidden from the user's view by the system, the content streams will continue to stream in their separate instantiations and remain synchronized.
[0069]
[0050] The displayed content streams are at the same or similar resolution as the native resolution of each video (e.g., 1920x1080 pixels), and the content streams (and / or the composite content stream) retain all of the functionality (playback options, etc.) of the user interface (e.g., web browser, video player, etc.). The system also provides additional controls such as, but not limited to, switching views from "video N" to N+1, switching views from "video N" to N-1, switching views from "video N" to N+ at a preset customizable rate (e.g., corresponding to a playlist), and zooming in on a portion of a video.
[0070]
[0051] The system can execute these functions in response to receiving a mouse click of the appropriate function by the user, a screen tap, dragging a section of the screen, or via keyboard shortcuts. As described below (e.g., in the playlist embodiment), these functions can also be triggered by pre-written commands stored in files accessible by the system.
[0071]
[0052] For example, if a user wishes to switch views from "Video N" to "Video N+1", the system hides the currently visible content stream (e.g., corresponding to one view) and displays "Video N+1" on the user's screen (e.g., user interface 102 (FIG. 1)). Because the current composite content stream is all loaded in memory (e.g., on a local device), the system can accomplish this switch at a fast rate that achieves the flicker fusion rate without synchronization issues or dropped frames. This process can be repeated for "Video N+2" or back to "Video N", N-1, etc., using controls provided by the system.
[0072]
[0053] In this example, when the content stream to be displayed becomes the last one ("Video #4") in the composite content stream corresponding to frame 500, the system receives a signal to play "Video #5" (the first content stream in the composite content stream corresponding to frame 550), and the system switches to a second instantiation of a user interface (e.g., a web browser, a video player, or a second instantiation of a standalone video player (which includes the composite content stream corresponding to frame 550)) and seamlessly displays "Video #5."
[0073]
[0054] When the last content stream of the composite content stream corresponding to frame 550 is reached, the system returns to the first instantiation (e.g., the first user interface) containing the composite content stream corresponding to frame 500. The process of switching content streams can be repeated in either direction under system and / or user control until the video ends.
[0074]
[0055] Note that content streams can be structured in any configuration in a composite content stream, for example N content streams horizontally or vertically, or a matrix of N content streams across and N down. Additionally or alternatively, the resolution of the content streams can be adjusted (either in pre-production or automatically by the system) to suit the bandwidth and capabilities of the server and player combination.
[0075] 6 is an illustration of a concatenated composite frame based on multiple content streams, according to one or more embodiments. For example, in some embodiments, a composite content stream may include multiple content streams for a media asset that are concatenated by appending one content stream to the end of another content stream, as shown in composite content stream 600. For example, each content stream of the second multiple content streams may be appended to one of the first multiple content streams.
[0076]
[0057] For example, there is no limit on the number of content streams that can be embedded in a single composite content stream (and / or the number of frames that can be embedded in a single composite frame), nor is there a limit on the number of composite content streams that can be loaded into separate instantiations of a user interface (e.g., a video playback system), although technical limitations may be imposed by server speed, transmission bandwidth, and / or user device memory. The system can alleviate some of the inherent memory / bandwidth limitations when instantiating more than a certain number of user interfaces simultaneously in memory (e.g., when processing each composite content stream).
[0077]
[0058] For example, as a non-limiting embodiment, if a computer / server imposes (e.g., based on technical limitations) a maximum of two instantiations of a user interface (e.g., a video player), each of which contains a composite content stream (e.g., each of which contains four individual content streams, as described above), there is a limit of eight content streams (e.g., corresponding to eight videos corresponding to eight views).
[0078]
[0059] Accordingly, to embed additional content streams within the two user interface instantiations (e.g., 16 content streams corresponding to 16 videos with different views), the system can concatenate the content streams. As shown in FIG. 6, "Video 9" is appended to the end of "Video 1". Using this method, the system can add additional content streams. For example, "Video 9+N" can be appended to the end of "Video 1+N". Moreover, the system can load each composite content stream into a separate instance of the user interface (e.g., a web browser or media player), synchronize the composite content streams so that they all play the same frame, and generate for display only the first frame of the top left video of the composite content stream 600 (e.g., including "Video 1" and "Video 9").
[0079]
[0060] Figure 7 is an illustration of a pair of concatenated composite frames based on multiple content streams, according to one or more embodiments. For example, upon receipt of a user request by the system, the system switches the display from "video 1 + video 9" to "video 2 + video 10". The system can repeat this process in either direction as many times as the user desires.
[0080]
[0061] When the displayed content stream becomes the last one in a given composite content stream sequence (e.g., in this example, composite content stream 700, "video 4-12"), the system switches the display to the first content stream in composite content stream 750 ("video 5 + video 13") and begins resynchronizing the timing of composite content stream 700. Accordingly, the timing pointer of composite content stream 700 is positioned to the same time location in the second half of the concatenated content streams of composite content stream 700.
[0081]
[0062] For example, if the media asset has a duration of 10 seconds, then all content streams have the same duration (e.g., 10 seconds). The system positions the pointer of the composite content stream 700 at the current display time (e.g., "current time" + 10 seconds) of the second half of the composite content stream 700 (e.g., the half corresponding to the added content stream). The system can perform this process in the background and out of the user's view in the user interface. In this way, the pointers are synchronized in time, and this synchronization is maintained to avoid the appearance of dropped frames when the system eventually accesses the second half of the content stream 700.
[0082]
[0063] For example, the system can load each composite content stream into a unique instantiation of the user interface (e.g., in the browser's internal video player). The system can synchronize all composite content streams in time. The system can then start playing all composite content streams and maintain the time synchronization (e.g., even if only a single content stream is visible). The system can display the first composite content stream (e.g., "Video #1") while hiding all other content streams and / or composite content streams. Upon receiving a user input (e.g., requesting a change of view and / or zoom), the system switches the display to reveal "Video N+1". Upon receiving a second user input (e.g., requesting a change of view and / or zoom), the system increments the number of the content stream to display. When the content stream to display becomes the last content stream of a given composite content stream, the system can trigger the display of the next content stream in the sequence. Accordingly, the system switches to the next user interface instantiation and displays the first content stream of composite content stream N+1. If there are no more composite content streams, the system returns to the first composite content stream. At the same time, the system resynchronizes the hidden composite content streams so that they are played back at the current time plus duration.
[0083]
[0064] FIG. 8 is an illustration of selecting an area of a frame to zoom, according to one or more embodiments. For example, in some embodiments, the system may allow a user to invoke a zoom function. The system may then determine a sequence of views to transition through when switching from a current view to a new view based on the total number of content streams available for the media asset to maintain the zoomed view. For example, the system may allow a user to select a level of zoom. The amount of zoom may vary, and the area manifested by the zoom (e.g., top left or bottom right) may be chosen by the user (e.g., mouse click, screen tap, voice command, etc.). These zoom areas may be configured in any manner, including multiple zoom factors, and may be located in any portion of the video, as shown in FIG. 8.
[0084]
[0065] However, because playback of a scene featured in a series of content streams may be captured by multiple content capture devices configured in any number of spatial arrangements, a traditional zoom operation, e.g., to the upper right, may be insufficient because the area of interest will necessarily shift as the user selects different viewing angles. With this in mind, the system may transition between a series of views, as illustrated in Figures 9A and 9B.
[0085] 9A is an illustration of determining a sequence of views to transition between when switching between views, according to one or more embodiments. For example, the system can receive a first user input, where the first user input selects a first view to present a media asset. The system can determine a current view in which the media asset is being presented. The system can determine a sequence of views to transition between when switching from the current view to the first view. The system can determine a content stream corresponding to each of the sequence of views.
[0086] For example, FIG. 9A illustrates a transition 900. In the example of FIG. 9A, the system may use media assets captured using 16 content capture devices arranged in a circle that capture a scene (e.g., three people sitting on chairs). In this case, the user wants to zoom into the upper left area (e.g., "Video 1") that is primarily occupied by a bald musician. When the next content stream in the sequence is displayed (e.g., while transitioning the view as described in FIGS. 11 and 12), the zoom area gradually moves (e.g., as shown in "Video 2"), and by the time the user interface displays "Video 8" (e.g., to 180 degrees of the circle), the zoom area is located in the upper right, maintaining the general location of the person originally selected.
[0087]
[0068] For example, the system can receive a first user input, where a first view is selected to present the media asset. The first view can include a particular viewing angle and / or zoom level. The system can then determine a current view and zoom level at which the media asset is currently presented. Based on the current view and zoom level, the system can determine a sequence of views and corresponding zoom levels to transition to when switching from the current view to the first view. The system can determine both the views and the zoom level for each view to maintain a seamless transition between the views. The system can then determine a content stream corresponding to each of the sequence of views. The system can then automatically transition through the content streams corresponding to the views, automatically adjusting the zoom level based on the zoom level determined for each view.
[0088]
[0069] Figure 9B illustrates a transition 900 and provides an exemplary description of the operations being performed to select a zoom level to achieve the effect of Figure 9A. For example, in the example of Figure 9B, the system may use media assets captured using 48 content capture devices arranged in a circle that capture a scene (e.g., three people sitting in chairs). To simplify the example, in this example, assume that the user can select from five zoom regions: top left, top right, bottom left, bottom right, and center. Note that these operations also apply to examples with additional regions and / or with any zoom area and magnification.
[0089]
[0070] As shown in transition 950, the system receives a selection of a first zoom area for a first content capture device (e.g., via user input). As shown, the first zoom covers 56.25% of the frame of "Video 1". In response to the system receiving a subsequent selection (e.g., via another user input), the system switches to a second content capture device (e.g., content capture device 1+N) and the zoom area is shifted to the second zoom area. As shown, the second content capture device represents the 24th content capture device of the 48 content capture devices. Thus, the view now appears to the right of the scene of "Video 2".
[0090]
[0071] To achieve this effect, the system uses a calculation based on a percentage of the entire frame. For example, in this case, the calculation for the horizontal (right) increment amount per content capture device corresponds to the percentage difference per content capture device. In this example, the system determines that the calculation is ((100%-56.25%) / 24)=1.8%.
[0091]
[0072] When the system reaches content capture device 24, the zoom position will be as in "Image 2". Completing the circuit, the increment is reversed by 1.8%, so that when content capture device 48 is reached, the zoom is in the same place as it started. The system can repeat this process for the transition from "Image 3" to "Image 4". On the other hand, no increment is needed if the central zoom area is selected, as shown in "Image 5".
[0092]
[0073] Figure 10 is an illustration of a playlist of a series of views, according to one or more embodiments. For example, in some embodiments, a system can receive a playlist, where the playlist includes preselected views for presenting a media asset. The system can then determine the views (and corresponding content streams for display) based on the playlist. For example, to provide a "director's cut" feature, the system can load a playlist that includes specific views and other characteristics (e.g., levels of magnification, zoom, etc.) that correspond to different time marks.
[0093] For example, the system may load a playlist 1000. The playlist 1000 may enable the system to use predefined controls for content stream features, including, but not limited to, rate of change of content stream selection, content stream selection direction (left / right), zoom functionality, pause / play, etc.
[0094]
[0075] Accordingly, a user can optionally view the functionality of the system without activating any controls. For example, the system can allow users to optionally record their own playback preferences and create their own "director's cut" to share with other viewers. This can be accomplished in many ways, one example being creating a text file with specific instructions that the system can load and execute, and storing it on a server or on the user's computer. For example, the system can receive a series of playlists, which include preselected views for presenting the media asset. The system can then determine a current view for presentation based on the playlist. In some embodiments, the system can monitor the user's viewing of the content stream during playback of the media asset. For example, the system can tag each frame with an indication that it was used by the user in the first composite content stream. The system can aggregate the tagged frames into a playback file. Moreover, the system can automatically compile and / or automatically share this file with other users (e.g., so that other users can view the media asset using the same content stream selection as the user).
[0095] 11 illustrates a flowchart of steps involved in providing rapid content switching for a media asset featuring multiple content streams distributed over a computer network, according to one or more embodiments. For example, a system can use process 1100 (e.g., as implemented on one or more system components) to provide rapid content switching.
[0096]
[0077] In step 1102, process 1100 receives a composite content stream (e.g., using one or more components described in system 400 (FIG. 4)). For example, the system can receive a first composite content stream based on a first composite frame and a second composite frame. For example, the first composite frame is based on a first frame set, the first frame set including a first frame from each of the first multiple content streams corresponding to a first time mark of each of the first multiple content streams. The second composite frame is based on a second frame set, the second frame set including a second frame from each of the first multiple content streams corresponding to a second time mark of each of the first multiple content streams. The first multiple content streams are for a media asset, and each content stream of the first multiple content streams corresponds to a respective view of a scene of the media asset.
[0097]
[0078] In step 1104, process 1100 processes the composite content stream for display (e.g., using one or more components described in system 400 (FIG. 4)). For example, the system may process the first composite content stream for display in a first user interface of a user device. For example, the system may process the composite content stream by selecting one of the views included in the composite content stream and generating only the frames corresponding to that view.
[0098]
[0079] For example, the system can receive a first user input, where the first user input selects a first view for presenting the media asset. The system can then determine that a first content stream of the first multiple content streams corresponds to the first view. In response to determining that the first content stream of the first multiple content streams corresponds to the first view, the system can determine a location in a composite frame of the first composite content stream that corresponds to a frame from the first content stream. The system can scale the location to fit a display area of a first user interface of the user device. For example, the system can scale the location to fit a display area of a first user interface of the user device by generating a frame from the first content stream for display to the user and not generating a frame from another content stream of the first multiple content stream for display to the user. The system can generate a location for display in the first user interface of the user device that is scaled to fit a display area of the first user interface of the user device.
[0099]
[0080] It is contemplated that the steps or descriptions of FIG. 11 can be used in any other embodiment of this disclosure. In addition, the steps and descriptions described in connection with FIG. 11 can be performed in alternative orders or in parallel to further the objectives of this disclosure. For example, each of these steps can be performed in any order, in parallel, or simultaneously to reduce lag and increase the speed of the system or method. Moreover, it is noted that any of the devices or equipment discussed in connection with FIGS. 1-10 and 12 can be used to perform one or more of the steps of FIG. 11.
[0100] 12 shows a flowchart of steps involved in generating a media asset featuring multiple content streams, according to one or more embodiments. For example, a system can use process 1200 (e.g., as implemented on one or more system components) to provide rapid content switching.
[0101]
[0082] In step 1202, process 1200 collects a first multiple content stream for the media asset (e.g., using one or more components described in system 400 (FIG. 4)). For example, the system may collect a first multiple content stream for the media asset, where each content stream of the first multiple content stream corresponds to a respective view of a scene of the media asset. The first multiple content streams may be stored at a remote location and / or transferred from a remote location. In some embodiments, each content stream may also have a corresponding audio track (e.g., captured with the same content capture device (e.g., via a microphone) as that content stream.
[0102] In step 1204, process 1200 retrieves a first set of frames (e.g., using one or more components described in system 400 (FIG. 4)). For example, the system can retrieve a first set of frames, where the first set of frames includes a first frame from each of the first multiple content streams that corresponds to a first time mark in each of the first multiple content streams.
[0103] In step 1206, process 1200 retrieves a second set of frames (e.g., using one or more components described in system 400 (FIG. 4)). For example, the system can retrieve a second set of frames, where the second set of frames includes a second frame from each of the first multiple content streams that corresponds to a second time mark in each of the first multiple content streams.
[0104]
[0085] In step 1208, process 1200 generates a first composite frame based on the first frame set (e.g., using one or more components described in system 400 (FIG. 4)). For example, the system may generate the first composite frame based on the first frame set. For example, the first composite frame may include respective portions corresponding to each frame of the first frame set (e.g., as shown in FIG. 5). For example, the first multiple content streams include four content streams and the first composite frame includes equal portions for the first frame from each of the first multiple content streams.
[0105]
[0086] In step 1210, process 1200 generates a second composite frame based on the second frame set (e.g., using one or more components described in system 400 (FIG. 4)). For example, the system may generate the second composite frame based on the second frame set. For example, the second composite frame may include respective portions corresponding to each frame of the second frame set (e.g., as shown in FIG. 5). For example, the second multiple content streams include four content streams and the first composite frame includes equal portions for the first frame from each of the first multiple content streams.
[0106]
[0087] In step 1212, process 1200 generates a first composite content stream (e.g., using one or more components described in system 400 (FIG. 4)). For example, the system may generate the first composite content stream based on the first composite frame and the second composite frame. In some embodiments (e.g., as described in FIG. 6), the first composite content stream may include a third multiple content stream for the media asset, where each content stream of the third multiple content stream corresponds to a respective view of a scene of the media asset, and where each content stream of the third multiple content stream is appended to one of the first multiple content streams.
[0107]
[0088] It is contemplated that the steps or descriptions of FIG. 12 can be used in any other embodiment of this disclosure. In addition, the steps and descriptions described in connection with FIG. 12 can be performed in alternative orders or in parallel to further the objectives of this disclosure. For example, each of these steps can be performed in any order, in parallel, or simultaneously to reduce lag and increase the speed of the system or method. Moreover, it is noted that any of the devices or equipment discussed in connection with FIGS. 1-11 can be used to perform one or more of the steps of FIG. 12.
[0108]
[0089] Figure 13 is an exemplary system for generating a media asset featuring multiple content streams over a large area, according to one or more embodiments. To achieve 360-degree camera coverage over a large area, such as a football field, an arena, a studio, or any indoor or outdoor environment, multiple cameras can be mounted on poles, on stands, hung from rafters, or otherwise mounted at similar heights in a circle, oval, or some other arrangement around the field, each camera having a lens of the same or similar focal length such that the field of view of each camera covers most of the field, as shown in Figure 13.
[0109]
[0090] In such cases, the system may mount all content capture devices at similar heights and have the same or similar focal lengths so that the transition effect remains smooth and continuous when switching video feeds (e.g., content streams). However, while the above-described content capture device setup results in a smooth, continuous rotation around the field with nearly the entire field visible from each camera, this arrangement presents other technical hurdles. Specifically, to cover the entire field, each camera would necessarily have to have a short focal length (e.g., characterized by a wide angle), and individual players would be too small for effective viewing.
[0110]
[0091] To overcome this technical hurdle, the system can focus on a specific area of view. For example, the action in a large field or arena is likely to be focused on a relatively small section of the field, so it is rarely worth recording the entire field at once. This is not an issue with traditional sporting event coverage using multiple cameras, because the cameras are not coordinated and each camera can independently zoom in on a portion of the field.
[0111]
[0092] However, to maintain coherence between the zoom settings and orientations of each of the multiple content capture devices and to achieve the rapid content switching playback effect described above, each content capture device may zoom, swivel, and / or tilt in a coordinated manner. For example, a content capture device that is physically closer to the action may require a short focal length, and a content capture device that is physically farther away may require a long focal length. In such cases, each content capture device may require movement independent of the others. Similarly, depending on the location of the subject (e.g., the corner of the filing cabinet where the action is taking place), the content capture devices may need to swivel and / or tilt relative to their location and / or independently of the other content capture devices.
[0112]
[0093] Figure 14 illustrates an exemplary content capture device for generating a media asset featuring multiple content streams, according to one or more embodiments. For example, Figure 14 may illustrate a mechanical means for following the action and maintaining coordinated zoom, pan, and / or tilt for a matrix of content capture devices (e.g., providing rapid content switching).
[0113] For example, a stadium, arena, sports ring, movie studio, or indoor or outdoor space of any size may be surrounded by a matrix of content capture devices mounted at a similar height and similar distance from the center of the pitch, arena, or sports ring (e.g., as described above). Additional content capture devices may be mounted above or below each other to achieve up / down control during playback in response to user input. In some embodiments, the shape and / or height of the matrix may be circular, elliptical, or any shape that corresponds to the needs of the user (e.g., director, viewer, etc.).
[0114]
[0095] Each content capture device may be mounted on a gimbal (or other mechanical means) with rotation capability (pan and tilt) in the X and Y axes. Each content capture device may employ a variable zoom lens. The pan and / or tilt settings of the gimbal and / or the focal length of the content capture device may be remotely controlled (e.g., via radio or other electromagnetic signals). Moreover, the zoom, pan and / or tilt settings of each content capture device may be determined automatically, as described above.
[0115] 15 is an exemplary system for generating a media asset featuring multiple content streams in a large area featuring pre-selected sections with desired fields of view, according to one or more embodiments. For example, when capturing a scene in a large setting where the main action moves over time to various locations in the environment, the system can follow the action by selecting the X, Y, Z coordinates of a single point in the playing area that corresponds to a point of interest (COI). The system can automatically modify the zoom, pan, and / or tilt settings of each content capture device to correspond to the COI (e.g., COI 1502).
[0116]
[0097] For example, as a media asset (e.g., a live recording of a match) progresses, the COI may move to different areas of the field. The COI may be tracked by a user and / or by an artificial intelligence model trained to follow the action. Additionally or alternatively, the COI may be based on the location of an element that correlates with the COI (e.g., RFID transmitters embedded in a player's ball, clothing and / or equipment). The element may transmit the X, Y and Z axes of the COI to a computer program via radio or other electromagnetic signals. For example, the Z axis may be zero (e.g., ground level) but may be changed if it is desired to track the ball during a kick or throw.
[0117]
[0098] As shown in Figure 15, the system can determine the COI, which is highlighted as a circled area in Figure 15. The COI can be a pre-selected section of any size that represents a desired field of view for all content capture devices when zoomed in on the scene.
[0118]
[0099] When a user selects a COI, the user can input the X, Y, (and Z) coordinates into the system via a user interface (e.g., with a mouse or similar input device) using an image of the viewing area (e.g., field) for reference. In the case of an artificial intelligence model, the model can be pre-trained to select the COI. If an RFID embodiment is used, the system can determine the COI using the actual location of the RFID chip and transmit the coordinates directly to the system.
[0119]
[0100] To coordinate the tilt / pan and focal length settings so that each content capture device maintains its relationship to the COI, each content capture device can independently adjust its focal length depending on its distance from the COI and reorient the gimbal pan / tilt so that the camera is pointed directly at the COI.
[0120] In some embodiments, the system may include a database of the X, Y, Z location of each content capture device to perform a series of trigonometric calculations that return the distance and angle of each content capture device relative to the COI. By combining two sets of coordinates (e.g., the content capture device and the COI), the system can use algorithms to calculate the X pan and Y tilt gimbal settings and focal length of each content capture device so that the content capture device is pointed directly at the COI and has a focal length proportional to the distance of the COI.
[0121]
[0102] The system can also generate automatic updates. For example, using wireless or other electromagnetic means, the system can send the unique focal length to all content capture devices in real time, and the content capture devices can adjust their zoom factors accordingly. Similarly, the X-axis pan setting and the Y-axis tilt setting can be sent to the gimbal, which will adjust its orientation.
[0122] 15, the content capture device 1504 is closest to the action and will have the shortest focal length (wide angle). Thus, the content capture device 1504 may have a gimbal pointing downward at an acute angle such that its center of view is directly aligned with the X, Y, Z coordinates of the COI 1502. The tilt orientation of each gimbal may be proportional to the distance of the content capture device's camera from the COI 1502 (e.g., the closer to the action, the greater the downward tilt). The content capture device directly adjacent to the content capture device 1504 may have a slightly longer focal length and slightly different tilt / pan settings, while the settings of the content capture device 1506 may feature, for example, a significantly longer focal length, a more extreme left pan, and a less extreme downtilt.
[0123]
[0104] Figure 16 shows an example diagram related to zoom calculations, according to one or more embodiments. For example, the focal length (zoom) setting of a content capture device can be calculated using trigonometric functions such that the focal length is directly proportional to the physical distance between the content capture device and the COI. Trigonometric calculations can be done in many ways, one example of which is shown in Figure 16, where a football field is represented by a series of equally spaced squares of any size.
[0124] In this example, the COI (e.g., COI 1502 (FIG. 15)) is located at an XY coordinate of (4,4) and the content capture device is located at an XY coordinate of (0,5). The distance between the content capture device and COI 1502 is the hypotenuse of the resulting triangle, i.e., (4 2 +1 2 ) 0.5 =4.13. The system can multiply this number by a predefined variable that is the same for all content capture devices to define the focal length setting of the zoom lens and the resulting field of view (e.g., the higher the variable, the higher the magnification).
[0125] To calculate pan (e.g., X, Y axis), the system can determine an angle θ, which represents the angle the gimbal should be panned (left or right) to bring the COI into the center of the content capture device's field of view. The system can determine the angle using the SIN of (1 / 4).
[0126]
[0107] Figure 17 shows an example diagram related to tilt calculation, according to one or more embodiments. To calculate the tile (e.g., X-axis), the system can use the height and distance (e.g., retrieved from a database) of the content capture device relative to the COI. The system can then calculate the downward tilt (θ) based on a TAN of (4.31 / 5) (height distance). Moreover, because the individual content capture devices are located in fairly close proximity to each other, slight changes in settings (relative to each other) can produce smooth variations in both zoom and orientation during playback.
[0127]
[0108] FIG. 18 illustrates an exemplary diagram related to the post-production of a media asset featuring multiple content streams, according to one or more embodiments. For example, the resulting zoom factor as described above is limited only by the optical capabilities of the zoom lens of the content capture device, which typically reaches 10x or more. For situations where a smaller zoom is desirable (e.g., a boxing ring instead of a football field), the system can forego the use of expensive gimbals and zoom lenses by deploying post-production digital zoom techniques without compromising the quality of the footage. During post-production, the COI can be dynamically chosen by the user, the model and / or the COI coordinates when integrating with the known positions of each content capture device. The system can then trigonometrically calculate the portion of the footage to crop. For example, as shown in FIG. 18, if the final 360-degree footage is to be streamed or transmitted at 1280 x 720 pixels (720HD) and the original footage was recorded at 5120 x 2880 pixels (5K), then in the post-production process, a digital zoom can be performed to a given portion of each image and that portion of the image can be rendered in 720HD using the trigonometric function solutions described above.
[0128]
[0109] The above-described embodiments of the present disclosure are presented for illustrative purposes, not for limitation, and the present disclosure is limited only by the claims that follow. Moreover, it should be noted that the features and limitations described in an embodiment may also be applied to any other embodiment herein, and that the flow charts or examples related to an embodiment may be combined with any other embodiment in an appropriate manner, performed in a different order, or performed in parallel. In addition, the systems and methods described herein may be performed in real time. It should also be noted that the above-described systems and / or methods may also be applied to or used in accordance with other systems and / or methods.
[0129]
[0110] The present technique will be better understood with reference to the enumerated embodiments below. 1. A method comprising: collecting first multiple content streams for a media asset, each content stream of the first multiple content streams corresponding to a respective view of a scene of the media asset; collecting a first set of frames, the first set of frames including a first frame from each of the first multiple content streams corresponding to a first time mark of each of the first multiple content streams; collecting a second set of frames, the second set of frames including a second frame from each of the first multiple content streams corresponding to a second time mark of each of the first multiple content streams; generating a first composite frame based on the first set of frames; generating a second composite frame based on the second set of frames; and generating a first composite content stream based on the first composite frames and the second composite frames. 2. A method comprising receiving a first composite content stream based on a first composite frame and a second composite frame, the first composite frame being based on a first frame set, the first frame set including a first frame from each of the first multiple content streams corresponding to a first time mark of each of the first multiple content streams, and the second composite frame being based on a second frame set, the second frame set including a second frame from each of the first multiple content streams corresponding to a second time mark of each of the first multiple content streams, the first multiple content streams being for a media asset, each content stream of the first multiple content streams corresponding to a respective view of a scene of the media asset, and processing the first composite content stream for display in a first user interface of a user device. 3. A method according to any of the preceding embodiments, further comprising: receiving a first user input, wherein the first user input selects a first view for presenting the media asset; determining that a first content stream of the first multiple content streams corresponds to the first view; in response to determining that the first content stream of the first multiple content streams corresponds to the first view, determining a location within a composite frame of the first composite content stream that corresponds to a frame from the first content stream; scaling the location to fit a display area of a first user interface of the user device; and generating for display in the first user interface of the user device, the location scaled to fit the display area of the first user interface of the user device. 4. A method as described in any of the preceding embodiments, wherein scaling the location to fit a display area of a first user interface of a user device includes generating frames from a first content stream for display to the user and not generating frames from other content streams of the first multiple content streams for display to the user. 5. The method of any of the preceding embodiments, further comprising: receiving a second composite content stream based on a third composite frame and a fourth composite frame, wherein the third composite frame is based on a third frame set, the third frame set including a first frame from each of the second multiple content streams corresponding to a first time mark of each of the second multiple content streams; the second composite frame is based on a second frame set, the second frame set including a second frame from each of the second multiple content streams corresponding to a second time mark of each of the second multiple content streams, the first multiple content streams being for a media asset, each content stream of the first multiple content streams corresponding to a respective view of a scene of the media asset; and processing the second composite content stream for display in a second user interface of the user device, wherein the second composite content stream is processed simultaneously with the first composite content stream. 6. A method according to any of the preceding embodiments, further comprising receiving a second user input, wherein the second user input selects a second view presenting the media asset, determining that a second content stream of the second multiple content streams corresponds to the second view, and replacing the first user interface with the second user interface in response to determining that the second content stream of the second multiple content streams corresponds to the second view. 7. The method of any of the preceding embodiments, wherein the first multiple content streams include four content streams and the first composite frame includes equal portions for the first frame from each of the first multiple content streams. 8. A method according to any of the preceding embodiments, wherein the first composite content stream includes a third multiple content stream for the media asset, each content stream of the third multiple content stream corresponding to a respective view of a scene of the media asset, and each content stream of the third multiple content stream being appended to one of the first multiple content streams. 9. The method of any of the preceding embodiments, further comprising receiving a playlist, the playlist including preselected views for presenting the media asset, and determining a current view for presentation based on the playlist. 10. A method according to any of the preceding embodiments, further comprising receiving a first user input, the first user input selecting a first view in which to present the media asset, determining a current view in which the media asset is currently presented, determining a sequence of views to transition to when switching from the current view to the first view, and determining a content stream corresponding to each of the sequence of views. 11. The method of any of the preceding embodiments, wherein the sequence of views to transition through when switching from a current view to a first view is based on a total number of content streams available for the media asset. 12. A tangible, non-transitory, machine-readable medium storing instructions that, when executed by a data processing apparatus, cause the data processing apparatus to perform operations including any of embodiments 1-11. 13. A system including one or more processors and a memory storing instructions, which, when executed by the processor, cause the processor to perform operations including any of embodiments 1 to 11. 14. A system including means for carrying out any of the first to eleventh embodiments.
Claims
1. 1. A system for providing rapid content switching of a media asset featuring multiple content streams delivered over a computer network, comprising: a cloud-based storage circuit configured to store a first number of content streams; 1. A cloud-based control circuit, comprising: retrieving the first multiple content streams for a media asset, each content stream of the first multiple content streams corresponding to a respective view of a scene of the media asset; retrieving a first set of frames, the first set of frames including a first frame from each of the first multiple content streams corresponding to a first time mark in each of the first multiple content streams; retrieving a second set of frames, the second set of frames including second frames from each of the first multiple content streams corresponding to a second time mark in each of the first multiple content streams; generating a first composite frame based on the first frame set; generating a second composite frame based on the second frame set; and generating a first composite content stream based on the first composite frame and the second composite frame; receiving a first user input, the first user input selecting a first view presenting the media asset for display on a first user interface of a user device; determining that a first content stream of the first multiple content streams corresponds to the first view; in response to determining that the first content stream of the first multiple content streams corresponds to the first view, determining a location within a composite frame of the first composite content stream that corresponds to a frame from the first content stream; scaling the location to fit a display area of the first user interface of the user device, the scaling including generating the frames from the first content stream for display to a user and not generating frames from other content streams of the first multiple content streams for display to the user; A control circuit configured to: input / output circuitry configured to generate for display, in the first user interface of the user device, the location scaled to fit the display area of the first user interface of the user device; Including, the system.
2. 1. A method for providing rapid content switching of a media asset featuring multiple content streams delivered over a computer network, comprising: receiving a first composite content stream based on the first composite frame and the second composite frame, the first composite frame is based on a first frame set, the first frame set including a first frame from each of the first multiple content streams corresponding to a first time mark in each of the first multiple content streams; the second composite frame is based on a second set of frames, the second set of frames including a second frame from each of the first multiple content streams corresponding to a second time mark in each of the first multiple content streams; the first multiple content streams are for a media asset, each content stream of the first multiple content streams corresponding to a respective view of a scene of the media asset; processing the first composite content stream for display at a first user interface of a user device; A method comprising:
3. receiving a first user input, the first user input selecting a first view for presenting the media asset; determining that a first content stream of the first multiple content streams corresponds to the first view; in response to determining that the first content stream of the first multiple content streams corresponds to the first view, determining a location within a composite frame of the first composite content stream that corresponds to a frame from the first content stream; Scaling the location to fit a display area of the first user interface of the user device; and generating for display in the first user interface of the user device, the location scaled to fit the display area of the first user interface of the user device; The method of claim 2 , further comprising:
4. 4. The method of claim 3, wherein scaling the location to fit the display area of the first user interface of the user device includes generating the frames from the first content stream for display to the user and not generating frames from other content streams of the first multiple content streams for display to the user.
5. receiving a second composite content stream based on the third composite frame and the fourth composite frame, the third composite frame is based on a third frame set, the third frame set including a first frame from each of the second multiple content streams corresponding to the first time mark of each of the second multiple content streams; the second composite frame is based on the second frame set, the second frame set including a second frame from each of the second multiple content streams corresponding to a second time mark in each of the second multiple content streams; the first multiple content streams for the media asset, each content stream of the first multiple content streams corresponding to a respective view of the scene of the media asset; processing the second composite content stream for display at a second user interface of a user device, the second composite content stream being processed simultaneously with the first composite content stream; The method of claim 2 , further comprising:
6. receiving a second user input, the second user input selecting a second view for presenting the media asset; determining that a second content stream of the second multiple content streams corresponds to the second view; in response to determining that the second content stream of the second multiple content streams corresponds to the second view, replacing the first user interface with the second user interface; The method of claim 5 further comprising:
7. 3. The method of claim 2, wherein the first multiple content streams include four content streams, and the first composite frame includes equal portions for the first frame from each of the first multiple content streams.
8. 3. The method of claim 2, wherein the first composite content stream includes a third multiple content stream for the media asset, each content stream of the third multiple content streams corresponding to a respective view of the scene of the media asset, and each content stream of the third multiple content streams is appended to one of the first multiple content streams.
9. receiving a playlist, the playlist including preselected views for presenting the media assets, the preselected views being automatically determined based on the first composite content stream; determining a current view for presentation based on the playlist; The method of claim 2 , further comprising:
10. receiving a first user input, the first user input selecting a first view for presenting the media asset; determining a current view and zoom level at which the media asset is currently presented; determining a sequence of views and corresponding zoom levels to transition through when switching from the current view to the first view; determining a content stream corresponding to each of the series of views; The method of claim 2 , further comprising:
11. The method of claim 10 , wherein the sequence of views to transition through when switching from the current view to the first view is based on a total number of content streams available for the media asset.
12. determining a composite audio track for presentation with the first composite content stream, the composite audio track including a first audio track corresponding to the first composite frame and a second audio track corresponding to the second composite frame, the first audio track being captured at a content capture device that captured the first set of frames, and the second audio track being captured at a content capture device that captured the second set of frames. The method of claim 2 , further comprising:
13. 1. A non-transitory computer-readable medium for providing rapid content switching of a media asset featuring multiple content streams delivered over a computer network, the instructions, when executed by one or more processors, performing the following steps: receiving a first composite content stream based on the first composite frame and the second composite frame, the first composite frame is based on a first frame set, the first frame set including a first frame from each of the first multiple content streams corresponding to a first time mark in each of the first multiple content streams; the second composite frame is based on a second set of frames, the second set of frames including a second frame from each of the first multiple content streams corresponding to a second time mark in each of the first multiple content streams; the first multiple content streams are for a media asset, each content stream of the first multiple content streams corresponding to a respective view of a scene of the media asset; processing the first composite content stream for display at a first user interface of a user device; A non-transitory computer-readable medium for causing a computer to perform operations including:
14. The instruction: receiving a first user input, the first user input selecting a first view for presenting the media asset; determining that a first content stream of the first multiple content streams corresponds to the first view; in response to determining that the first content stream of the first multiple content streams corresponds to the first view, determining a location within a composite frame of the first composite content stream that corresponds to a frame from the first content stream; Scaling the location to fit a display area of the first user interface of the user device; and generating for display in the first user interface of the user device, the location scaled to fit the display area of the first user interface of the user device; The non-transitory computer-readable medium of claim 13 , further comprising:
15. 15. The non-transitory computer-readable medium of claim 14, wherein scaling the location to fit the display area of the first user interface of the user device includes generating the frames from the first content stream for display to a user and not generating frames from other content streams of the first multiple content streams for display to the user.
16. The instruction: receiving a second composite content stream based on the third composite frame and the fourth composite frame, the third composite frame is based on a third frame set, the third frame set including a first frame from each of the second multiple content streams corresponding to the first time mark of each of the second multiple content streams; the second composite frame is based on the second frame set, the second frame set including a second frame from each of the second multiple content streams corresponding to a second time mark in each of the second multiple content streams; the first multiple content streams for the media asset, each content stream of the first multiple content streams corresponding to a respective view of the scene of the media asset; processing the second composite content stream for display at a second user interface of a user device, the second composite content stream being processed simultaneously with the first composite content stream; The non-transitory computer-readable medium of claim 13 , further comprising:
17. The instruction: receiving a second user input, the second user input selecting a second view for presenting the media asset; determining that a second content stream of the second multiple content streams corresponds to the second view; in response to determining that the second content stream of the second multiple content streams corresponds to the second view, replacing the first user interface with the second user interface; 20. The non-transitory computer-readable medium of claim 16, further comprising:
18. 14. The non-transitory computer-readable medium of claim 13, wherein the first multiple content streams include four content streams and the first composite frame includes equal portions for the first frame from each of the first multiple content streams.
19. 14. The non-transitory computer-readable medium of claim 13, wherein the first composite content stream includes a third multiple content stream for the media asset, each content stream of the third multiple content streams corresponding to a respective view of the scene of the media asset, and each content stream of the third multiple content streams is appended to one of the first multiple content streams.
20. The instruction: receiving a playlist, the playlist including preselected views presenting the media assets; determining a current view for presentation based on the playlist; The non-transitory computer-readable medium of claim 13 , further comprising:
21. The instruction: receiving a first user input, the first user input selecting a first view for presenting the media asset; determining a current view in which the media asset is currently presented; determining a set of views to transition through when switching from the current view to the first view, the set of views to transition through when switching from the current view to the first view being based on a total number of content streams available for the media asset; determining a content stream corresponding to each of the series of views; The non-transitory computer-readable medium of claim 13 , further comprising:
Citation Information
Patent Citations
Methods and apparatus for encoding multiple video signals into a single encoded video signal and methods and apparatus for decoding such encoded video signals
JP2008533890A
Surround video playback
US20130039632A1
Aligning videos representing different viewpoints
US20150222815A1
Image display method and image display device
WO2018199052A1
